[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:49 dist-test-slave-39l8 krb5kdc[9022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:49 dist-test-slave-39l8 krb5kdc[9022](info): set up 2 sockets
Aug 06 23:56:49 dist-test-slave-39l8 krb5kdc[9022](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 23:56:50.388377 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:56:50 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:56:50.504004 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38553
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:50.611425 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:50.611652 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:50.611689 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:50.615660 9044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:50.615707 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:50.615726 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:50.615741 9044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:50.615793 9044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:50.620568 9044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:50.621447 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:50.622354 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:50.630105 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:50.630129 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:50.630133 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:50.630357 9044 server_base.cc:1034] running on GCE node
I20260806 23:56:50.630962 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:50.631889 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:50.633049 9044 hybrid_clock.cc:648] HybridClock initialized: now 1786060610633056 us; error 120 us; skew 500 ppm
Aug 06 23:56:50 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:50.635588 9044 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:56:50.636276 9044 webserver.cc:460] Webserver started at http://127.8.204.126:34511/ using document root <none> and password file <none>
I20260806 23:56:50.636732 9044 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:50.636782 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:50.636920 9044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:50.638777 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "5b853aae42494915b85df55c736e4b70"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "be75cfd87981f5934a221db698960c56"
server_key_iv: "360070a88c463705def24f63d9366464"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.639335 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "5b853aae42494915b85df55c736e4b70"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "be75cfd87981f5934a221db698960c56"
server_key_iv: "360070a88c463705def24f63d9366464"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.642588 9044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:56:50.644913 9070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:50.645998 9044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:50.646092 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "5b853aae42494915b85df55c736e4b70"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "be75cfd87981f5934a221db698960c56"
server_key_iv: "360070a88c463705def24f63d9366464"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.646167 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:50.659358 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:50.661741 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:50.661868 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:50.668200 9044 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:38553
I20260806 23:56:50.668219 9148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:38553 every 8 connection(s)
I20260806 23:56:50.669059 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:50.669756 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9044
I20260806 23:56:50.669922 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:56:50.671128 9009 external_mini_cluster.cc:1428] Setting key 945fe5f253abdfb96008379cb2bc267c
I20260806 23:56:50.671811 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:50.677723 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70: Bootstrap starting.
I20260806 23:56:50.679981 9149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:50.680691 9149 log.cc:824] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:50.682744 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70: No bootstrap required, opened a new log
I20260806 23:56:50.686349 9149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } }
I20260806 23:56:50.686566 9149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:50.686648 9149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b853aae42494915b85df55c736e4b70, State: Initialized, Role: FOLLOWER
I20260806 23:56:50.687147 9149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } }
I20260806 23:56:50.687312 9149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:50.687388 9149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:50.687484 9149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 23:56:50 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:50.688349 9149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } }
I20260806 23:56:50.688716 9149 leader_election.cc:302] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b853aae42494915b85df55c736e4b70; no voters:
I20260806 23:56:50.689100 9149 leader_election.cc:288] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:50.689230 9166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:50.689435 9166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [term 1 LEADER]: Becoming Leader. State: Replica: 5b853aae42494915b85df55c736e4b70, State: Running, Role: LEADER
I20260806 23:56:50.689734 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } }
I20260806 23:56:50.690423 9149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:50.691293 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b853aae42494915b85df55c736e4b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } } }
I20260806 23:56:50.691432 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:50.691807 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b853aae42494915b85df55c736e4b70. Latest consensus state: current_term: 1 leader_uuid: "5b853aae42494915b85df55c736e4b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b853aae42494915b85df55c736e4b70" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38553 } } }
I20260806 23:56:50.691910 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:50.692297 9177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:50.694375 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:50.673940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.8.204.126:38553)
0806 23:56:50.674291 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:50.674308 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:50.675365 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 23:56:50.678337 (+ 2972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:50.678351 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:50.678649 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:50.679029 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:50.681193 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.682542 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.683690 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.684362 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.684522 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:50.688299 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:50.688306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:50.688323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:50.688355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:50.690400 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:50.691050 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:50.691058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:50.691067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:50.691209 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:50.691684 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:50.691695 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:50.691697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:50.692252 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:50.692453 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:50.693287 (+ 834us) server_negotiation.cc:299] Negotiation successful
0806 23:56:50.693491 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":111,"threads_started":1}
I20260806 23:56:50.695343 9177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:50.699903 9177 catalog_manager.cc:1347] Generated new cluster ID: 075a229596844cb99b92c41355d66687
I20260806 23:56:50.699963 9177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:50.714758 9177 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:50.715910 9177 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:50.724771 9177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b853aae42494915b85df55c736e4b70: Generated new TSK 0
I20260806 23:56:50.725364 9177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:56:50.753047 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:50.858213 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:50.858420 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:50.858456 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:50.862166 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:50.862215 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:50.862265 9197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:56:50.866732 9197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:50.867573 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:50.868462 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:50.877499 9207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:50.877511 9208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:50.877528 9210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:50.877566 9197 server_base.cc:1034] running on GCE node
I20260806 23:56:50.878096 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:50.878665 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:50.879899 9197 hybrid_clock.cc:648] HybridClock initialized: now 1786060610879866 us; error 37 us; skew 500 ppm
Aug 06 23:56:50 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:50.882321 9197 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:56:50.883126 9197 webserver.cc:460] Webserver started at http://127.8.204.65:45633/ using document root <none> and password file <none>
I20260806 23:56:50.883600 9197 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:50.883656 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:50.883807 9197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:50.885668 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "9ee975c3bfc04a69990ba6ce3c78bf6b"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "1b0dc74772ef3a600b45ebb073afef64"
server_key_iv: "844ead15adec7b88d49e1e3fdc1d3b05"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.886170 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "9ee975c3bfc04a69990ba6ce3c78bf6b"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "1b0dc74772ef3a600b45ebb073afef64"
server_key_iv: "844ead15adec7b88d49e1e3fdc1d3b05"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.889635 9197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260806 23:56:50.892001 9218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:50.893123 9197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:56:50.893265 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "9ee975c3bfc04a69990ba6ce3c78bf6b"
format_stamp: "Formatted at 2026-08-06 23:56:50 on dist-test-slave-39l8"
server_key: "1b0dc74772ef3a600b45ebb073afef64"
server_key_iv: "844ead15adec7b88d49e1e3fdc1d3b05"
server_key_version: "encryptionkey@0"
I20260806 23:56:50.893361 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:50.908018 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:50.910573 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:50.910806 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:50.911345 9197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:50.912137 9197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:50.912210 9197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:50.912284 9197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:50.912325 9197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:50.920414 9197 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:34857
I20260806 23:56:50.920491 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:34857 every 8 connection(s)
I20260806 23:56:50.921219 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:50.928627 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9197
I20260806 23:56:50.928727 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:50.928968 9009 external_mini_cluster.cc:1428] Setting key 3127ed6d58c5104a216fc19a5985c54e
Aug 06 23:56:50 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:50.934954 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:50.922966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:57351 (local address 127.8.204.126:38553)
0806 23:56:50.923102 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:50.923106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:50.923856 (+ 750us) server_negotiation.cc:407] Connection header received
0806 23:56:50.925159 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:50.925163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:50.925224 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:50.925324 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:50.926395 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.926995 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.928058 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.928306 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.928415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:50.931225 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:50.931229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:50.931232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:50.931256 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:50.932937 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:50.933420 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:50.933426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:50.933429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:50.933528 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:50.933849 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:50.933854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:50.933856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:50.934026 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:50.934107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:50.934550 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 23:56:50.934699 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:56:50.935509 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:50.923324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38553 (local address 127.8.204.65:57351)
0806 23:56:50.923720 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:50.923757 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:50.924940 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:50.925501 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:50.925507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:50.925819 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:50.926246 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:50.926258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.927131 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.927134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:50.927937 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:50.927946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:50.928482 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:50.928486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:50.928647 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:50.929307 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:50.929322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:50.931068 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:50.933062 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:50.933066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:50.933077 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:50.933321 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:50.933633 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:50.933636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:50.933637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:50.933755 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:50.934123 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:50.934127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:50.934327 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 23:56:50.934465 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 23:56:50.934675 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260806 23:56:50.936790 9380 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38553
I20260806 23:56:50.937050 9380 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:50.937662 9380 heartbeater.cc:511] Master 127.8.204.126:38553 requested a full tablet report, sending...
I20260806 23:56:50.939248 9094 ts_manager.cc:194] Registered new tserver with Master: 9ee975c3bfc04a69990ba6ce3c78bf6b (127.8.204.65:34857)
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:56:50.940392 9094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:57351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:56:50.955646 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:51.065383 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:51.065635 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:51.065706 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:51.069658 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:51.069738 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:51.069826 9385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:56:51.075088 9385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:51.076074 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:51.077052 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:51.086412 9385 server_base.cc:1034] running on GCE node
W20260806 23:56:51.086463 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:51.086354 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:51.086356 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:51.087042 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:51.087659 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:51.088816 9385 hybrid_clock.cc:648] HybridClock initialized: now 1786060611088792 us; error 36 us; skew 500 ppm
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:51.091706 9385 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:56:51.092494 9385 webserver.cc:460] Webserver started at http://127.8.204.66:42951/ using document root <none> and password file <none>
I20260806 23:56:51.093022 9385 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:51.093113 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:51.093341 9385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:51.095283 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "21e5233d6c044606aebc3176e5bde446"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "6e92773d62c0fabbe88e86b48f7146cc"
server_key_iv: "a5498cc9552a44bd5eba1524288b7a9e"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.095809 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "21e5233d6c044606aebc3176e5bde446"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "6e92773d62c0fabbe88e86b48f7146cc"
server_key_iv: "a5498cc9552a44bd5eba1524288b7a9e"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.099284 9385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:51.101531 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.102835 9385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:51.102967 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "21e5233d6c044606aebc3176e5bde446"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "6e92773d62c0fabbe88e86b48f7146cc"
server_key_iv: "a5498cc9552a44bd5eba1524288b7a9e"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.103065 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:51.116144 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:51.118655 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:51.118851 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:51.119410 9385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:51.120182 9385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:51.120262 9385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.120332 9385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:51.120374 9385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.128927 9385 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44825
I20260806 23:56:51.128974 9559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44825 every 8 connection(s)
I20260806 23:56:51.129752 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:51.131273 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9385
I20260806 23:56:51.131403 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:51.131678 9009 external_mini_cluster.cc:1428] Setting key 44b85d1748ead091c2a4ac9ea55b6ce6
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:56:51.143401 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.131484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:59727 (local address 127.8.204.126:38553)
0806 23:56:51.131627 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:51.131632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:51.132441 (+ 809us) server_negotiation.cc:407] Connection header received
0806 23:56:51.133881 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:51.133887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:51.133947 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:51.134040 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:51.135162 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.135664 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.136705 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.137058 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.137184 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.139721 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:51.139725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:51.139727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:51.139750 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:51.141192 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.141788 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.141791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.141792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.141862 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.142263 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.142265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.142266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.142422 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:51.142492 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:51.143191 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 23:56:51.143291 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:56:51.144173 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.131819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38553 (local address 127.8.204.66:59727)
0806 23:56:51.132291 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:51.132340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:51.133685 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:51.134195 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:51.134201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:51.134506 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:51.134991 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:51.135004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.135806 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.135809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:51.136567 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:51.136576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.137213 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.137216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:51.137342 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.137817 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:51.137831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:51.139547 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:51.141316 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:51.141325 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:51.141343 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:51.141692 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:51.141957 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:51.141963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:51.141966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:51.142162 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:51.142549 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:51.142555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:51.142854 (+ 299us) client_negotiation.cc:769] Sending connection context
0806 23:56:51.143139 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 23:56:51.143424 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260806 23:56:51.145390 9560 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38553
I20260806 23:56:51.145627 9560 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:51.146153 9560 heartbeater.cc:511] Master 127.8.204.126:38553 requested a full tablet report, sending...
I20260806 23:56:51.147308 9094 ts_manager.cc:194] Registered new tserver with Master: 21e5233d6c044606aebc3176e5bde446 (127.8.204.66:44825)
I20260806 23:56:51.147842 9094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:59727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:56:51.159922 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:51.266908 9567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:51.267107 9567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:51.267143 9567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:51.270759 9567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:51.270805 9567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:51.270859 9567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:56:51.275600 9567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:51.276437 9567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:51.277243 9567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:51.285941 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:51.285979 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:51.285961 9576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:51.286152 9567 server_base.cc:1034] running on GCE node
I20260806 23:56:51.286549 9567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:51.287264 9567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:51.288444 9567 hybrid_clock.cc:648] HybridClock initialized: now 1786060611288431 us; error 31 us; skew 500 ppm
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:51.291011 9567 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:56:51.292122 9567 webserver.cc:460] Webserver started at http://127.8.204.67:38711/ using document root <none> and password file <none>
I20260806 23:56:51.292704 9567 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:51.292793 9567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:51.293011 9567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:51.294862 9567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "913d627905244bfe9e525dcc41930ad8"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "660565b49b728671d8f6ebee0741211e"
server_key_iv: "ab608e9a64be8486aa5e2beb06842111"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.295464 9567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "913d627905244bfe9e525dcc41930ad8"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "660565b49b728671d8f6ebee0741211e"
server_key_iv: "ab608e9a64be8486aa5e2beb06842111"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.298861 9567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:51.301272 9590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.302376 9567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:51.302501 9567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "913d627905244bfe9e525dcc41930ad8"
format_stamp: "Formatted at 2026-08-06 23:56:51 on dist-test-slave-39l8"
server_key: "660565b49b728671d8f6ebee0741211e"
server_key_iv: "ab608e9a64be8486aa5e2beb06842111"
server_key_version: "encryptionkey@0"
I20260806 23:56:51.302597 9567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:51.320919 9567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:51.323410 9567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:51.323604 9567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:51.324263 9567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:51.325126 9567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:51.325202 9567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.325290 9567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:51.325341 9567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:51.337167 9567 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:41059
I20260806 23:56:51.337199 9741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:41059 every 8 connection(s)
I20260806 23:56:51.337898 9567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:51.346144 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9567
I20260806 23:56:51.346262 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:51.346504 9009 external_mini_cluster.cc:1428] Setting key 4c2f4f9eb158ac5bf2dcc1c42d6b0b34
I20260806 23:56:51.350440 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.339615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:38021 (local address 127.8.204.126:38553)
0806 23:56:51.339761 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:51.339764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:51.340511 (+ 747us) server_negotiation.cc:407] Connection header received
0806 23:56:51.341668 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:51.341670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:51.341714 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:51.341784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:51.342744 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.343220 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.344194 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.344390 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.344463 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.346754 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:51.346758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:51.346761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:51.346790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:51.348279 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.348890 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.348895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.348897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.348982 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.349333 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.349337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.349339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.349508 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:51.349636 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:51.350137 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 23:56:51.350276 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:56:51.351150 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.339923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38553 (local address 127.8.204.67:38021)
0806 23:56:51.340361 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:51.340400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:51.341469 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:51.341922 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:51.341928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:51.342209 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:51.342590 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:51.342601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.343322 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.343324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:51.344080 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:51.344088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.344509 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.344511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:51.344644 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.345111 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:51.345125 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:51.346547 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:51.348391 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:51.348397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:51.348409 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:51.348793 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:51.349093 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:51.349096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:51.349098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:51.349237 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:51.349617 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:51.349624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:51.349907 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 23:56:51.350080 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 23:56:51.350384 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260806 23:56:51.352128 9742 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38553
I20260806 23:56:51.352344 9742 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:51.352797 9742 heartbeater.cc:511] Master 127.8.204.126:38553 requested a full tablet report, sending...
I20260806 23:56:51.353732 9094 ts_manager.cc:194] Registered new tserver with Master: 913d627905244bfe9e525dcc41930ad8 (127.8.204.67:41059)
I20260806 23:56:51.354223 9094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:38021
I20260806 23:56:51.360499 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:51.385864 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.377537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.8.204.65:34857)
0806 23:56:51.377859 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:51.377869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:51.377889 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:56:51.377954 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:51.377958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:51.378125 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:51.378233 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:51.378875 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.379663 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.380437 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.380744 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.380820 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.382827 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:51.382835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:51.382845 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:51.382866 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:51.384359 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.384717 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.384721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.384722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.384797 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.385015 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.385018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.385019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.385227 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:51.385375 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:51.385506 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 23:56:51.385666 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":131,"threads_started":1}
W20260806 23:56:51.386674 9257 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45530
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:51.411065 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:51.403322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.8.204.65:34857)
0806 23:56:51.403493 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:51.403497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:51.403620 (+ 123us) server_negotiation.cc:407] Connection header received
0806 23:56:51.403772 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:51.403781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:51.403817 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:51.403896 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:51.404506 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.404967 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.405820 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:51.406154 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:51.406281 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:51.408339 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:51.408342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:51.408344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:51.408364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:51.409517 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.409890 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.409893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.409894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.409964 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:51.410190 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:51.410192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:51.410193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:51.410344 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:51.410455 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:51.410687 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 23:56:51.410876 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 23:56:51.411998 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9197
I20260806 23:56:51.425249 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9385
I20260806 23:56:51.433246 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9567
I20260806 23:56:51.441684 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9044
2026-08-06T23:56:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2201 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9762](info): set up 2 sockets
Aug 06 23:56:51 dist-test-slave-39l8 krb5kdc[9762](info): commencing operation
krb5kdc: starting...
W20260806 23:56:52.594730 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:56:52 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:56:52.710666 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:32987
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:52.819223 9780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:52.819450 9780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:52.819487 9780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:52.823351 9780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:52.823400 9780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:52.823417 9780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:52.823433 9780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:52.823448 9780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:52.828271 9780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:52.829176 9780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:52.830120 9780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:52.838114 9791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:52.838126 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:52.838148 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:52.838379 9780 server_base.cc:1034] running on GCE node
I20260806 23:56:52.839064 9780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:52.840106 9780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:52.841288 9780 hybrid_clock.cc:648] HybridClock initialized: now 1786060612841260 us; error 45 us; skew 500 ppm
Aug 06 23:56:52 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060612, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:52.847802 9780 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:56:52.848680 9780 webserver.cc:460] Webserver started at http://127.8.204.126:44021/ using document root <none> and password file <none>
I20260806 23:56:52.849241 9780 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:52.849331 9780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:52.849557 9780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:52.851608 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "c03c6a32fb73493db8e782a482b8d6c3"
format_stamp: "Formatted at 2026-08-06 23:56:52 on dist-test-slave-39l8"
server_key: "521a28023dcd80e62753f5ae96528507"
server_key_iv: "e3fe7dd2d24232a5a585dcca558cb007"
server_key_version: "encryptionkey@0"
I20260806 23:56:52.852185 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "c03c6a32fb73493db8e782a482b8d6c3"
format_stamp: "Formatted at 2026-08-06 23:56:52 on dist-test-slave-39l8"
server_key: "521a28023dcd80e62753f5ae96528507"
server_key_iv: "e3fe7dd2d24232a5a585dcca558cb007"
server_key_version: "encryptionkey@0"
I20260806 23:56:52.855685 9780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:52.857908 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:52.859110 9780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:52.859274 9780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "c03c6a32fb73493db8e782a482b8d6c3"
format_stamp: "Formatted at 2026-08-06 23:56:52 on dist-test-slave-39l8"
server_key: "521a28023dcd80e62753f5ae96528507"
server_key_iv: "e3fe7dd2d24232a5a585dcca558cb007"
server_key_version: "encryptionkey@0"
I20260806 23:56:52.859380 9780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:52.877499 9780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:52.880100 9780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:52.880311 9780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:52.887209 9780 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:32987
I20260806 23:56:52.887226 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:32987 every 8 connection(s)
I20260806 23:56:52.888093 9780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:52.890672 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:52.895784 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: Bootstrap starting.
I20260806 23:56:52.896628 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9780
I20260806 23:56:52.896744 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:56:52.896984 9009 external_mini_cluster.cc:1428] Setting key 7830022817e7aacc0d79df84bc78af2d
I20260806 23:56:52.898317 9873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:52.899240 9873 log.cc:824] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:52.901269 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: No bootstrap required, opened a new log
I20260806 23:56:52.905143 9873 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } }
I20260806 23:56:52.905391 9873 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:52.905438 9873 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03c6a32fb73493db8e782a482b8d6c3, State: Initialized, Role: FOLLOWER
Aug 06 23:56:52 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:52.905893 9873 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } }
I20260806 23:56:52.906060 9873 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:52.906129 9873 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:52.906229 9873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:52.907308 9873 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } }
I20260806 23:56:52.907648 9873 leader_election.cc:302] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03c6a32fb73493db8e782a482b8d6c3; no voters:
I20260806 23:56:52.907946 9873 leader_election.cc:288] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:52.908068 9879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:52.908294 9879 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [term 1 LEADER]: Becoming Leader. State: Replica: c03c6a32fb73493db8e782a482b8d6c3, State: Running, Role: LEADER
I20260806 23:56:52.908653 9879 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } }
I20260806 23:56:52.909278 9873 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:52.910864 9882 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03c6a32fb73493db8e782a482b8d6c3. Latest consensus state: current_term: 1 leader_uuid: "c03c6a32fb73493db8e782a482b8d6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } } }
I20260806 23:56:52.910898 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03c6a32fb73493db8e782a482b8d6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03c6a32fb73493db8e782a482b8d6c3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 32987 } } }
I20260806 23:56:52.910977 9882 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:52.911005 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:52.912153 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:52.898110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.8.204.126:32987)
0806 23:56:52.898576 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:52.898597 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:52.898640 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:56:52.899459 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:52.899483 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:52.899821 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:52.900154 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:52.900820 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:52.902182 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:52.903064 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:52.903724 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:52.903890 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:52.906335 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:52.906340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:52.906370 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:52.906400 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:52.908388 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:52.908921 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:52.908927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:52.908932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:52.909065 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:52.909538 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:52.909542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:52.909543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:52.909996 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:52.910200 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:52.910881 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 23:56:52.911216 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
W20260806 23:56:52.914747 9897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:52.914831 9897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:52.914883 9890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:52.915688 9890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:52.921228 9890 catalog_manager.cc:1347] Generated new cluster ID: e3851b65ed5a4ddd991ef7cb12b6e469
I20260806 23:56:52.921310 9890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:52.948570 9890 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:52.949478 9890 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:52.955341 9890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03c6a32fb73493db8e782a482b8d6c3: Generated new TSK 0
I20260806 23:56:52.955964 9890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:56:52.987325 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:53.096907 9908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:53.097141 9908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:53.097183 9908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:53.101138 9908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:53.101193 9908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:53.101248 9908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:56:53.106392 9908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:53.107462 9908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:53.108474 9908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:53.118256 9916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:53.118317 9908 server_base.cc:1034] running on GCE node
W20260806 23:56:53.118271 9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:53.118301 9918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:53.118940 9908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:53.119549 9908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:53.120745 9908 hybrid_clock.cc:648] HybridClock initialized: now 1786060613120722 us; error 46 us; skew 500 ppm
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:53.123427 9908 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:56:53.124228 9908 webserver.cc:460] Webserver started at http://127.8.204.65:45883/ using document root <none> and password file <none>
I20260806 23:56:53.124796 9908 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:53.124888 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:53.125119 9908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:53.127125 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "e1371354f6914bfea9fbb3c96128de07"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "08a84b5c584d6d190f21506094acbe90"
server_key_iv: "b532963323f20d917e800949343ae258"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.127677 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "e1371354f6914bfea9fbb3c96128de07"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "08a84b5c584d6d190f21506094acbe90"
server_key_iv: "b532963323f20d917e800949343ae258"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.131165 9908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:53.133476 9930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.134609 9908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:53.134770 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "e1371354f6914bfea9fbb3c96128de07"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "08a84b5c584d6d190f21506094acbe90"
server_key_iv: "b532963323f20d917e800949343ae258"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.134882 9908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:53.148312 9908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:53.151016 9908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:53.151201 9908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:53.151791 9908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:53.152684 9908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:53.152731 9908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.152769 9908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:53.152784 9908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.163228 9908 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:38151
I20260806 23:56:53.163316 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:38151 every 8 connection(s)
I20260806 23:56:53.164145 9908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:53.173508 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 9908
I20260806 23:56:53.173600 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:53.173965 9009 external_mini_cluster.cc:1428] Setting key 2282617672674733250b7a4abe8694ba
I20260806 23:56:53.177371 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.165890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35679 (local address 127.8.204.126:32987)
0806 23:56:53.166036 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.166040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.167048 (+ 1008us) server_negotiation.cc:407] Connection header received
0806 23:56:53.167922 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.167925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.167970 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.168042 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.169166 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.169708 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.170841 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.171065 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.171143 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.173457 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.173460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.173462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.173485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.175256 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.175864 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.175867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.175868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.175942 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.176341 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.176343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.176344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.176502 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.176575 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.177150 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.177253 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 23:56:53.178236 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.166240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:32987 (local address 127.8.204.65:35679)
0806 23:56:53.166905 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.166944 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.167739 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.168204 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.168210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.168581 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:53.169032 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.169045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.169813 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.169815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.170669 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.170679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.171195 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.171198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.171331 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.171835 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:53.171850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:53.173293 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:53.175372 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.175379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.175395 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.175761 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.176044 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.176050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.176054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.176235 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.176605 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:53.176613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:53.176914 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.177111 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.177456 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":100,"threads_started":1}
I20260806 23:56:53.179425 10082 heartbeater.cc:347] Connected to a master server at 127.8.204.126:32987
I20260806 23:56:53.179663 10082 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:53.180116 10082 heartbeater.cc:511] Master 127.8.204.126:32987 requested a full tablet report, sending...
I20260806 23:56:53.181675 9828 ts_manager.cc:194] Registered new tserver with Master: e1371354f6914bfea9fbb3c96128de07 (127.8.204.65:38151)
I20260806 23:56:53.182827 9828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:35679
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:56:53.201440 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:53.310647 10089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:53.310914 10089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:53.310984 10089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:53.315203 10089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:53.315279 10089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:53.315362 10089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:56:53.320420 10089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:53.321377 10089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:53.322338 10089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:53.331594 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:53.331676 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:53.331606 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:53.331830 10089 server_base.cc:1034] running on GCE node
I20260806 23:56:53.332247 10089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:53.332859 10089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:53.334044 10089 hybrid_clock.cc:648] HybridClock initialized: now 1786060613334030 us; error 33 us; skew 500 ppm
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:53.336728 10089 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:56:53.337545 10089 webserver.cc:460] Webserver started at http://127.8.204.66:34233/ using document root <none> and password file <none>
I20260806 23:56:53.338120 10089 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:53.338212 10089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:53.338508 10089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:53.340566 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "2cab6d2d3c14c6e5bef496d8577ed4c4"
server_key_iv: "135550dc8e0ca208fc0964b26641d82f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.341120 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "2cab6d2d3c14c6e5bef496d8577ed4c4"
server_key_iv: "135550dc8e0ca208fc0964b26641d82f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.344559 10089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 23:56:53.346935 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.348024 10089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:53.348170 10089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "2cab6d2d3c14c6e5bef496d8577ed4c4"
server_key_iv: "135550dc8e0ca208fc0964b26641d82f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.348297 10089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:53.358690 10089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:53.361222 10089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:53.361403 10089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:53.361943 10089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:53.362815 10089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:53.362890 10089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.362947 10089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:53.362985 10089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.371634 10089 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:35905
I20260806 23:56:53.371662 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:35905 every 8 connection(s)
I20260806 23:56:53.372447 10089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:53.376971 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 10089
I20260806 23:56:53.377086 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:53.377370 9009 external_mini_cluster.cc:1428] Setting key 06814707163eeccf94debcf27d54feee
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:53.385571 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.374141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:60965 (local address 127.8.204.126:32987)
0806 23:56:53.374284 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.374288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.375032 (+ 744us) server_negotiation.cc:407] Connection header received
0806 23:56:53.375774 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.375777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.375820 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.375889 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.376937 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.377719 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.379176 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.379563 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.379686 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.381891 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.381895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.381897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.381917 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.383410 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.384038 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.384040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.384041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.384122 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.384487 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.384489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.384490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.384669 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.384742 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.385366 (+ 624us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.385456 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:56:53.386667 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.374451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:32987 (local address 127.8.204.66:60965)
0806 23:56:53.374893 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.374928 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.375578 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.376026 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.376031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.376355 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:53.376789 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.376801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.377863 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.377881 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.379049 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.379058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.379714 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.379717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.379846 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.380418 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:53.380434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:53.381734 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:53.383533 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.383542 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.383557 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.383937 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.384213 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.384216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.384218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.384394 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.384782 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:53.384792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:53.385101 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.385325 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.385669 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260806 23:56:53.387900 10275 heartbeater.cc:347] Connected to a master server at 127.8.204.126:32987
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:56:53.388139 10275 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:53.388589 10275 heartbeater.cc:511] Master 127.8.204.126:32987 requested a full tablet report, sending...
I20260806 23:56:53.389611 9828 ts_manager.cc:194] Registered new tserver with Master: 87bb0efc281e4d4da131f2a1c1ccaaf8 (127.8.204.66:35905)
I20260806 23:56:53.390246 9828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:60965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:56:53.404866 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:53.513881 10283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:53.514123 10283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:53.514164 10283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:53.517948 10283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:53.517997 10283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:53.518066 10283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:56:53.522915 10283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:32987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:53.523777 10283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:53.524631 10283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:53.533769 10294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:53.533802 10289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:53.533879 10288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:53.534158 10283 server_base.cc:1034] running on GCE node
I20260806 23:56:53.534546 10283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:53.535147 10283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:53.536326 10283 hybrid_clock.cc:648] HybridClock initialized: now 1786060613536313 us; error 23 us; skew 500 ppm
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:53.539108 10283 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:56:53.539965 10283 webserver.cc:460] Webserver started at http://127.8.204.67:34995/ using document root <none> and password file <none>
I20260806 23:56:53.540517 10283 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:53.540575 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:53.540736 10283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:53.542526 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "47490bf5bdc74175aaed04fb3f4e6360"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "a13ce824ff938372f71da7bf58539a3b"
server_key_iv: "266a12515ddb2badd18f5bcdf6bcd96f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.543076 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "47490bf5bdc74175aaed04fb3f4e6360"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "a13ce824ff938372f71da7bf58539a3b"
server_key_iv: "266a12515ddb2badd18f5bcdf6bcd96f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.546417 10283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:53.548650 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.549777 10283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:53.549906 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "47490bf5bdc74175aaed04fb3f4e6360"
format_stamp: "Formatted at 2026-08-06 23:56:53 on dist-test-slave-39l8"
server_key: "a13ce824ff938372f71da7bf58539a3b"
server_key_iv: "266a12515ddb2badd18f5bcdf6bcd96f"
server_key_version: "encryptionkey@0"
I20260806 23:56:53.549999 10283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:53.565577 10283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:53.568457 10283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:53.568591 10283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:53.569054 10283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:53.569828 10283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:53.569875 10283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.569912 10283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:53.569933 10283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:53.578831 10283 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:32965
I20260806 23:56:53.578867 10466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:32965 every 8 connection(s)
I20260806 23:56:53.579644 10283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:53.580747 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 10283
I20260806 23:56:53.580839 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:53.581090 9009 external_mini_cluster.cc:1428] Setting key 8b16c20ed5b9a958dd378d957279b011
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:53.591717 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.581392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:45639 (local address 127.8.204.126:32987)
0806 23:56:53.581561 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.581566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.582501 (+ 935us) server_negotiation.cc:407] Connection header received
0806 23:56:53.583468 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.583470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.583511 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.583573 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.584548 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.585041 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.586092 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.586285 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.586374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.588621 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.588624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.588626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.588666 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.590095 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.590529 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.590532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.590533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.590601 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.590893 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.590896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.590897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.591063 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.591135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.591525 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.591609 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:56:53.592407 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.581750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:32987 (local address 127.8.204.67:45639)
0806 23:56:53.582342 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.582401 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.583305 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.583691 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.583696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.583998 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:53.584437 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.584449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.585145 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.585148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.585979 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.585987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.586425 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.586428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.586587 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.587110 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:53.587125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:53.588466 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:53.590193 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.590197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.590206 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.590446 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.590678 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:53.590681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:53.590682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:53.590815 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:53.591148 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:53.591152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:53.591337 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.591464 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.591669 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":171,"threads_started":1}
I20260806 23:56:53.593503 10467 heartbeater.cc:347] Connected to a master server at 127.8.204.126:32987
I20260806 23:56:53.593750 10467 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:53.594326 10467 heartbeater.cc:511] Master 127.8.204.126:32987 requested a full tablet report, sending...
I20260806 23:56:53.595377 9828 ts_manager.cc:194] Registered new tserver with Master: 47490bf5bdc74175aaed04fb3f4e6360 (127.8.204.67:32965)
I20260806 23:56:53.595909 9828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:45639
I20260806 23:56:53.606406 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:53.616258 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.609659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.8.204.126:32987)
0806 23:56:53.609783 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.609787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.610044 (+ 257us) server_negotiation.cc:407] Connection header received
0806 23:56:53.610137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.610139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.610176 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.610254 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.611067 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.611553 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.612328 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.612525 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.612618 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.613836 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.613839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.613841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.613857 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.615017 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.615360 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.615362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.615363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.615428 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.615635 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.615637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.615638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.615818 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.615915 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.616076 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.616156 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:56:53.620921 9828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:53.624032 9828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:53.637931 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.632177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:38151 (local address 127.0.0.1:45874)
0806 23:56:53.633165 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.633178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.633262 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.633944 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.633947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.633959 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:53.634033 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.634041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.635356 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.635360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.636574 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.636585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637376 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.637380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.637479 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.637515 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.637666 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.637786 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":39,"thread_start_us":93,"threads_started":1}
I20260806 23:56:53.637941 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.632622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.8.204.65:38151)
0806 23:56:53.633510 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.633521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.633540 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:56:53.633624 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.633629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.633803 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.633956 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:53.634158 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.635226 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.636718 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637251 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.637375 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.637523 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.637698 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.637791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":101,"threads_started":1}
I20260806 23:56:53.638088 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.631785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:35905 (local address 127.0.0.1:35286)
0806 23:56:53.632224 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.632268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.632388 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.633476 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.633482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.633505 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:53.633637 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.633644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.635269 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.635272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.636625 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.636637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637716 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.637720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.637806 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.637827 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.637892 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.637957 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":84,"thread_start_us":85,"threads_started":1}
I20260806 23:56:53.638334 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.631852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:32965 (local address 127.0.0.1:41712)
0806 23:56:53.633069 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.633082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.633177 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.633864 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.633866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.633881 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:53.633991 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.633997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.635045 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.635051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.636625 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.636637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637903 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.637906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.638039 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.638097 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.638167 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.638225 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":109,"thread_start_us":103,"threads_started":1}
I20260806 23:56:53.638633 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.632510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.8.204.66:35905)
0806 23:56:53.632958 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.632971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.632995 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:56:53.633075 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.633093 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.633300 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.633784 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:53.633956 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.635134 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.636774 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637573 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.638188 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.638317 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.638368 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.638461 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":117,"threads_started":1}
I20260806 23:56:53.638909 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.632679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.8.204.67:32965)
0806 23:56:53.633025 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.633035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.633273 (+ 238us) server_negotiation.cc:407] Connection header received
0806 23:56:53.633357 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.633362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.633565 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.633716 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:53.634124 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.634920 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.636774 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.637573 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.637691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.637833 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.638617 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.638741 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":142,"threads_started":1}
I20260806 23:56:53.641319 9997 tablet_service.cc:1467] Processing CreateTablet for tablet f98967c98ed44162957336be3fd43140 (DEFAULT_TABLE table=test-table [id=28a89a872745401d91a67b0de9bcd22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:53.641626 10377 tablet_service.cc:1467] Processing CreateTablet for tablet f98967c98ed44162957336be3fd43140 (DEFAULT_TABLE table=test-table [id=28a89a872745401d91a67b0de9bcd22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:53.641626 10177 tablet_service.cc:1467] Processing CreateTablet for tablet f98967c98ed44162957336be3fd43140 (DEFAULT_TABLE table=test-table [id=28a89a872745401d91a67b0de9bcd22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:53.642343 9997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98967c98ed44162957336be3fd43140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:53.642621 10377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98967c98ed44162957336be3fd43140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:53.642621 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98967c98ed44162957336be3fd43140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:53.648231 10497 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Bootstrap starting.
I20260806 23:56:53.649415 10499 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: Bootstrap starting.
I20260806 23:56:53.650126 10500 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: Bootstrap starting.
I20260806 23:56:53.651464 10497 tablet_bootstrap.cc:654] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:53.652683 10500 tablet_bootstrap.cc:654] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:53.652856 10499 tablet_bootstrap.cc:654] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:53.652848 10497 log.cc:824] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:53.653497 10500 log.cc:824] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:53.653750 10499 log.cc:824] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:53.655056 10497 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: No bootstrap required, opened a new log
I20260806 23:56:53.655259 10497 ts_tablet_manager.cc:1397] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:56:53.655400 10500 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: No bootstrap required, opened a new log
I20260806 23:56:53.655570 10500 ts_tablet_manager.cc:1397] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:56:53.658990 10497 raft_consensus.cc:348] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.658990 10500 raft_consensus.cc:348] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.659260 10497 raft_consensus.cc:374] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:53.659260 10500 raft_consensus.cc:374] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:53.659351 10497 raft_consensus.cc:729] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1371354f6914bfea9fbb3c96128de07, State: Initialized, Role: FOLLOWER
I20260806 23:56:53.659351 10500 raft_consensus.cc:729] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47490bf5bdc74175aaed04fb3f4e6360, State: Initialized, Role: FOLLOWER
I20260806 23:56:53.659811 10497 consensus_queue.cc:260] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.659811 10500 consensus_queue.cc:260] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.660720 10467 heartbeater.cc:503] Master 127.8.204.126:32987 was elected leader, sending a full tablet report...
I20260806 23:56:53.660800 10500 ts_tablet_manager.cc:1428] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:53.660800 10497 ts_tablet_manager.cc:1428] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:53.660909 10499 tablet_bootstrap.cc:492] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: No bootstrap required, opened a new log
I20260806 23:56:53.661067 10499 ts_tablet_manager.cc:1397] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260806 23:56:53.661345 10082 heartbeater.cc:503] Master 127.8.204.126:32987 was elected leader, sending a full tablet report...
I20260806 23:56:53.664108 10499 raft_consensus.cc:348] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.664331 10499 raft_consensus.cc:374] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:53.664391 10499 raft_consensus.cc:729] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bb0efc281e4d4da131f2a1c1ccaaf8, State: Initialized, Role: FOLLOWER
I20260806 23:56:53.664770 10499 consensus_queue.cc:260] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
W20260806 23:56:53.665254 10084 tablet.cc:2369] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:53.665359 10275 heartbeater.cc:503] Master 127.8.204.126:32987 was elected leader, sending a full tablet report...
I20260806 23:56:53.665654 10499 ts_tablet_manager.cc:1428] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:53.682612 10508 raft_consensus.cc:482] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:53.682804 10508 raft_consensus.cc:504] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.683715 10508 leader_election.cc:288] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47490bf5bdc74175aaed04fb3f4e6360 (127.8.204.67:32965), e1371354f6914bfea9fbb3c96128de07 (127.8.204.65:38151)
I20260806 23:56:53.687392 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.683901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:32965 (local address 127.8.204.66:35841)
0806 23:56:53.684023 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.684039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.684134 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.684426 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.684428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.684450 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:53.684599 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.684606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.685411 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.685414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.686234 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.686244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.686967 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.686970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.687094 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.687116 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.687193 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.687267 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 23:56:53.687449 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.683966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:35841 (local address 127.8.204.67:32965)
0806 23:56:53.684074 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.684079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.684095 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:56:53.684254 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.684257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.684303 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.684388 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:53.684716 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.685301 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.686362 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.686855 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.686945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.686982 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.687232 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.687307 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:53.687492 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.684047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:36713 (local address 127.8.204.65:38151)
0806 23:56:53.684149 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.684152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.684431 (+ 279us) server_negotiation.cc:407] Connection header received
0806 23:56:53.684532 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.684535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.684598 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.684683 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:53.684961 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.685457 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.686516 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.686961 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.687073 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.687121 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.687313 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.687376 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 23:56:53.687714 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.683908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:38151 (local address 127.8.204.66:36713)
0806 23:56:53.684332 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:53.684347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:53.684433 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:53.684717 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:53.684719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:53.684737 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:53.684847 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.684853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.685574 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.685577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.686389 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:53.686399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.687073 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.687076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:53.687151 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.687197 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 23:56:53.687273 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 23:56:53.687336 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":109,"thread_start_us":111,"threads_started":1}
I20260806 23:56:53.688014 10405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98967c98ed44162957336be3fd43140" candidate_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47490bf5bdc74175aaed04fb3f4e6360" is_pre_election: true
I20260806 23:56:53.688261 10405 raft_consensus.cc:2454] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87bb0efc281e4d4da131f2a1c1ccaaf8 in term 0.
I20260806 23:56:53.688261 10021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98967c98ed44162957336be3fd43140" candidate_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1371354f6914bfea9fbb3c96128de07" is_pre_election: true
I20260806 23:56:53.688526 10021 raft_consensus.cc:2454] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87bb0efc281e4d4da131f2a1c1ccaaf8 in term 0.
I20260806 23:56:53.688798 10115 leader_election.cc:302] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47490bf5bdc74175aaed04fb3f4e6360, 87bb0efc281e4d4da131f2a1c1ccaaf8; no voters:
I20260806 23:56:53.689057 10508 raft_consensus.cc:2781] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:53.689144 10508 raft_consensus.cc:482] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:53.689193 10508 raft_consensus.cc:3037] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:53.690155 10508 raft_consensus.cc:504] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.690481 10508 leader_election.cc:288] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [CANDIDATE]: Term 1 election: Requested vote from peers 47490bf5bdc74175aaed04fb3f4e6360 (127.8.204.67:32965), e1371354f6914bfea9fbb3c96128de07 (127.8.204.65:38151)
I20260806 23:56:53.690799 10405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98967c98ed44162957336be3fd43140" candidate_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47490bf5bdc74175aaed04fb3f4e6360"
I20260806 23:56:53.690901 10021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98967c98ed44162957336be3fd43140" candidate_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1371354f6914bfea9fbb3c96128de07"
I20260806 23:56:53.690905 10405 raft_consensus.cc:3037] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:53.691016 10021 raft_consensus.cc:3037] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:53.691927 10405 raft_consensus.cc:2454] T f98967c98ed44162957336be3fd43140 P 47490bf5bdc74175aaed04fb3f4e6360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87bb0efc281e4d4da131f2a1c1ccaaf8 in term 1.
I20260806 23:56:53.692013 10021 raft_consensus.cc:2454] T f98967c98ed44162957336be3fd43140 P e1371354f6914bfea9fbb3c96128de07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87bb0efc281e4d4da131f2a1c1ccaaf8 in term 1.
I20260806 23:56:53.692260 10115 leader_election.cc:302] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47490bf5bdc74175aaed04fb3f4e6360, 87bb0efc281e4d4da131f2a1c1ccaaf8; no voters:
I20260806 23:56:53.692453 10508 raft_consensus.cc:2781] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:53.692656 10508 raft_consensus.cc:686] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [term 1 LEADER]: Becoming Leader. State: Replica: 87bb0efc281e4d4da131f2a1c1ccaaf8, State: Running, Role: LEADER
I20260806 23:56:53.692952 10508 consensus_queue.cc:237] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } }
I20260806 23:56:53.695804 9825 catalog_manager.cc:5495] T f98967c98ed44162957336be3fd43140 P 87bb0efc281e4d4da131f2a1c1ccaaf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87bb0efc281e4d4da131f2a1c1ccaaf8 (127.8.204.66). New cstate: current_term: 1 leader_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87bb0efc281e4d4da131f2a1c1ccaaf8" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 35905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47490bf5bdc74175aaed04fb3f4e6360" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 32965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1371354f6914bfea9fbb3c96128de07" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 38151 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:53.718309 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.710253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.8.204.65:38151)
0806 23:56:53.710403 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.710408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.710614 (+ 206us) server_negotiation.cc:407] Connection header received
0806 23:56:53.710801 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.710803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.710874 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.710949 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.711664 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.712142 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.712895 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.713083 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.713153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.715408 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.715417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.715431 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.715452 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.716957 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.717326 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.717331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.717332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.717411 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.717628 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.717630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.717631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.717836 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.717986 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.718113 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.718206 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:53.746076 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.737751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.8.204.65:38151)
0806 23:56:53.737878 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.737881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.737983 (+ 102us) server_negotiation.cc:407] Connection header received
0806 23:56:53.738140 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.738142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.738173 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.738241 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.738906 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.739422 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.740176 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.740377 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.740450 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.742880 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.742885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.742891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.742917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.744627 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.745003 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.745005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.745006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.745076 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.745310 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.745312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.745313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.745457 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.745570 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.745767 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.745855 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[9762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:53.770881 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:53.762888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.8.204.65:38151)
0806 23:56:53.763028 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:53.763033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:53.763218 (+ 185us) server_negotiation.cc:407] Connection header received
0806 23:56:53.763392 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:53.763394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:53.763431 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:53.763515 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:53.764186 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.764657 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.765479 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:53.765683 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:53.765788 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:53.768012 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:53.768021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:53.768023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:53.768045 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:53.769215 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.769651 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.769655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.769656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.769738 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:53.770020 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:53.770023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:53.770025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:53.770193 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:53.770333 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:53.770594 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 23:56:53.770726 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 23:56:53.772199 9972 tablet_service.cc:2908] Rejecting scan request for tablet f98967c98ed44162957336be3fd43140: Uninitialized: clean time has not yet been initialized
I20260806 23:56:53.774333 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9908
I20260806 23:56:53.787920 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 10089
I20260806 23:56:53.795590 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 10283
I20260806 23:56:53.804786 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 9780
2026-08-06T23:56:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2362 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[10535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[10535](info): set up 2 sockets
Aug 06 23:56:53 dist-test-slave-39l8 krb5kdc[10535](info): commencing operation
krb5kdc: starting...
W20260806 23:56:54.964500 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:56:55.081621 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:39051
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:55.188323 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:55.188562 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:55.188601 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:55.192444 10558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:55.192488 10558 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:56:55.192505 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:55.192521 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:55.192538 10558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:55.192551 10558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:55.197299 10558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:39051
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:55.198167 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:55.199162 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:55.207158 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:55.207201 10565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.207196 10558 server_base.cc:1034] running on GCE node
W20260806 23:56:55.207362 10567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.207865 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:55.208845 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:55.210002 10558 hybrid_clock.cc:648] HybridClock initialized: now 1786060615209984 us; error 29 us; skew 500 ppm
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:55.212692 10558 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:56:55.213438 10558 webserver.cc:460] Webserver started at http://127.8.204.126:45713/ using document root <none> and password file <none>
I20260806 23:56:55.213922 10558 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:55.213972 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:55.214125 10558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:55.216043 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "214cbc80d57c421f826e42aab04d6da7"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "e04555ea14718aa61dcd9443b2a79ec1"
server_key_iv: "cc04ffc244687dc1981b6c858aacd203"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.216586 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "214cbc80d57c421f826e42aab04d6da7"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "e04555ea14718aa61dcd9443b2a79ec1"
server_key_iv: "cc04ffc244687dc1981b6c858aacd203"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.219970 10558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:55.222325 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.223482 10558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:55.223601 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "214cbc80d57c421f826e42aab04d6da7"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "e04555ea14718aa61dcd9443b2a79ec1"
server_key_iv: "cc04ffc244687dc1981b6c858aacd203"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.223695 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:55.241616 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:55.244486 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:55.244691 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:55.252784 10558 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:39051
I20260806 23:56:55.252789 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:39051 every 8 connection(s)
I20260806 23:56:55.253738 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:55.256476 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:55.257467 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 10558
I20260806 23:56:55.257587 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:56:55.257889 9009 external_mini_cluster.cc:1428] Setting key ca6f7fc03e5ba08c37e7be69988db4eb
I20260806 23:56:55.262924 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Bootstrap starting.
I20260806 23:56:55.265553 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:55.266363 10658 log.cc:824] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:55.268380 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: No bootstrap required, opened a new log
I20260806 23:56:55.271345 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } }
I20260806 23:56:55.271571 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:55.271646 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cbc80d57c421f826e42aab04d6da7, State: Initialized, Role: FOLLOWER
I20260806 23:56:55.272095 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.259241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.8.204.126:39051)
0806 23:56:55.259695 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.259711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.259747 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:55.260358 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.260374 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.260696 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.260958 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.261821 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.262789 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.263850 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.264284 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.264416 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.267012 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.267017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.267029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.267056 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.269248 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.269738 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.269743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.269748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.269856 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.270146 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.270150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.270152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.270531 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.270763 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.271037 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.271305 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":164,"threads_started":1}
I20260806 23:56:55.272116 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } }
I20260806 23:56:55.272289 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:55.272353 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:55.272449 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:55.273392 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } }
I20260806 23:56:55.273738 10658 leader_election.cc:302] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214cbc80d57c421f826e42aab04d6da7; no voters:
I20260806 23:56:55.274066 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:55.274152 10667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:55.274353 10667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [term 1 LEADER]: Becoming Leader. State: Replica: 214cbc80d57c421f826e42aab04d6da7, State: Running, Role: LEADER
I20260806 23:56:55.274665 10667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } }
I20260806 23:56:55.275215 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:55.275731 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214cbc80d57c421f826e42aab04d6da7. Latest consensus state: current_term: 1 leader_uuid: "214cbc80d57c421f826e42aab04d6da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } } }
I20260806 23:56:55.275851 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:55.276620 10668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214cbc80d57c421f826e42aab04d6da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cbc80d57c421f826e42aab04d6da7" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 39051 } } }
I20260806 23:56:55.276721 10668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:55.277524 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:55.280705 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:55.285650 10673 catalog_manager.cc:1347] Generated new cluster ID: de28c45d12cc4647b9f9ee7d3ddd3e11
I20260806 23:56:55.285712 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:55.302757 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:55.303642 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:55.314659 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Generated new TSK 0
I20260806 23:56:55.315300 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:56:55.346886 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:39051
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:55.457332 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:55.457623 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:55.457702 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:55.461608 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:55.461684 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:55.461743 10694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:55.461813 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:56:55.466398 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:55.467453 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:55.468426 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:55.477631 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:55.477654 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.477777 10694 server_base.cc:1034] running on GCE node
W20260806 23:56:55.477880 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.478335 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:55.478978 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:55.480176 10694 hybrid_clock.cc:648] HybridClock initialized: now 1786060615480158 us; error 28 us; skew 500 ppm
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:55.482693 10694 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:56:55.483554 10694 webserver.cc:460] Webserver started at http://127.8.204.65:38001/ using document root <none> and password file <none>
I20260806 23:56:55.484105 10694 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:55.484197 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:55.484403 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:55.486294 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "7009d7aed7714e2dbb34fb7220152a5f"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "eb5062a39ca1d6dce7eeab2486995321"
server_key_iv: "19671eba079e5f877d60b7f804739ba3"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.486931 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "7009d7aed7714e2dbb34fb7220152a5f"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "eb5062a39ca1d6dce7eeab2486995321"
server_key_iv: "19671eba079e5f877d60b7f804739ba3"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.490465 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:55.492902 10716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.494083 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:55.494181 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "7009d7aed7714e2dbb34fb7220152a5f"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "eb5062a39ca1d6dce7eeab2486995321"
server_key_iv: "19671eba079e5f877d60b7f804739ba3"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.494256 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:55.514348 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:55.516866 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:55.517067 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:55.518491 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:55.518564 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.518633 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:55.518688 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:55.529508 10694 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:44103
I20260806 23:56:55.529582 10872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:44103 every 8 connection(s)
I20260806 23:56:55.530639 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:55.532709 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.519151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:60991 (local address 127.8.204.126:39051)
0806 23:56:55.519308 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.519315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.520733 (+ 1418us) server_negotiation.cc:407] Connection header received
0806 23:56:55.522043 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.522046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.522103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.522184 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.523817 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.524451 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.525663 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.525917 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.526018 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.528625 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.528629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.528632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.528661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.530763 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.531325 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.531330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.531332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.531427 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.531792 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.531794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.531796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.531984 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.532068 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.532479 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.532592 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:55.533470 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 10694
I20260806 23:56:55.533586 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:55.533625 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.519988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.65:60991)
0806 23:56:55.520475 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.520563 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.521789 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.522375 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.522387 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.522948 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.523639 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.523662 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.524601 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.524605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.525540 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.525551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.526064 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.526068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.526219 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.526762 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.526778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.528415 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.530897 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.530903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.530916 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.531198 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.531532 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.531537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.531542 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.531694 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.532081 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.532085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.532282 (+ 197us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.532425 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.532640 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
I20260806 23:56:55.533917 9009 external_mini_cluster.cc:1428] Setting key c17a4889b68bfcf6cdc4810eacb3790b
I20260806 23:56:55.535116 10873 heartbeater.cc:347] Connected to a master server at 127.8.204.126:39051
I20260806 23:56:55.535269 10873 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:55.535828 10873 heartbeater.cc:511] Master 127.8.204.126:39051 requested a full tablet report, sending...
I20260806 23:56:55.537628 10595 ts_manager.cc:194] Registered new tserver with Master: 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103)
I20260806 23:56:55.538946 10595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:60991
I20260806 23:56:55.545660 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.537527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.65:38521)
0806 23:56:55.537844 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.537860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.537978 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.538226 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.538232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.538455 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.538611 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.538621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.539970 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.539973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.540784 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.540795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.541420 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.541422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.541493 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.541963 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.541973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.542273 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.544310 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.544313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.544314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.544493 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.544792 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.544795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.544796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.544865 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.545305 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.545307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.545374 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.545446 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.545536 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
I20260806 23:56:55.545823 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.537346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:38521 (local address 127.8.204.126:39051)
0806 23:56:55.537443 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.537447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.537944 (+ 497us) server_negotiation.cc:407] Connection header received
0806 23:56:55.538054 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.538056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.538102 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.538186 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.538776 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.539820 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.540909 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.541291 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.541421 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.542418 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.542425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.542428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.542457 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.544184 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.544593 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.544598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.544602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.544693 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.544959 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.544964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.544967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.545195 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.545313 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.545540 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.545657 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:56:55.563890 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:39051
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:55.667876 10884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:55.668120 10884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:55.668177 10884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:55.672013 10884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:55.672058 10884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:55.672112 10884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:55.672143 10884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:56:55.677649 10884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:55.678613 10884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:55.679632 10884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:55.688864 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.688895 10884 server_base.cc:1034] running on GCE node
W20260806 23:56:55.688864 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:55.688974 10895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.689677 10884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:55.690456 10884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:55.691630 10884 hybrid_clock.cc:648] HybridClock initialized: now 1786060615691586 us; error 63 us; skew 500 ppm
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:55.694308 10884 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:56:55.695150 10884 webserver.cc:460] Webserver started at http://127.8.204.66:38687/ using document root <none> and password file <none>
I20260806 23:56:55.695684 10884 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:55.695744 10884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:55.695976 10884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:55.698091 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "51b079c298454454bd05c1549234025d"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "17d1ca4abf3c3bdb6aab705dc26ee093"
server_key_iv: "75b217b32c1f76dc07796a2b3c2ad030"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.698688 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "51b079c298454454bd05c1549234025d"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "17d1ca4abf3c3bdb6aab705dc26ee093"
server_key_iv: "75b217b32c1f76dc07796a2b3c2ad030"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.702255 10884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:55.704519 10904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.705682 10884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:55.705816 10884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "51b079c298454454bd05c1549234025d"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "17d1ca4abf3c3bdb6aab705dc26ee093"
server_key_iv: "75b217b32c1f76dc07796a2b3c2ad030"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.705932 10884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:55.724545 10884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:55.727305 10884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:55.727509 10884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:55.729005 10884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:55.729076 10884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.729142 10884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:55.729184 10884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:55.739436 10884 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:45009
I20260806 23:56:55.740474 10884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:55.740530 11063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:45009 every 8 connection(s)
I20260806 23:56:55.743003 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.729734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:57425 (local address 127.8.204.126:39051)
0806 23:56:55.729891 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.729895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.731199 (+ 1304us) server_negotiation.cc:407] Connection header received
0806 23:56:55.732392 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.732396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.732461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.732563 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.733648 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.734566 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.735930 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.736148 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.736258 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.739488 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.739495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.739497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.739527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.741194 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.741726 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.741729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.741730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.741804 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.742051 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.742053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.742054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.742212 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.742296 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.742775 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.742890 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:56:55.743690 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.730607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.66:57425)
0806 23:56:55.731071 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.731114 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.732139 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.732765 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.732771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.733111 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.733519 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.733530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.734741 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.734746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.735823 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.735837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.736290 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.736293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.736461 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.737094 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.737116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.739270 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.741297 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.741302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.741315 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.741607 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.741860 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.741862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.741863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.741971 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.742312 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.742316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.742498 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.742635 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.742961 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":158,"threads_started":1}
I20260806 23:56:55.744793 11064 heartbeater.cc:347] Connected to a master server at 127.8.204.126:39051
I20260806 23:56:55.744939 11064 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:55.745388 11064 heartbeater.cc:511] Master 127.8.204.126:39051 requested a full tablet report, sending...
I20260806 23:56:55.746364 10597 ts_manager.cc:194] Registered new tserver with Master: 51b079c298454454bd05c1549234025d (127.8.204.66:45009)
I20260806 23:56:55.747087 10597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:57425
I20260806 23:56:55.750430 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 10884
I20260806 23:56:55.750554 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:55.750916 9009 external_mini_cluster.cc:1428] Setting key 3dfbe060951611f140815a77e844cab9
I20260806 23:56:55.753398 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.746786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45171 (local address 127.8.204.126:39051)
0806 23:56:55.746901 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.746905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.747049 (+ 144us) server_negotiation.cc:407] Connection header received
0806 23:56:55.747160 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.747162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.747208 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.747291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.747707 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.748137 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.748924 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.749130 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.749235 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.750194 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.750198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.750201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.750232 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.751718 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.752193 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.752199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.752202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.752283 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.752569 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.752574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.752575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.752791 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.752900 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.753160 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.753262 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:56:55.753410 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.746686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.66:45171)
0806 23:56:55.746963 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.746982 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.747086 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.747305 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.747308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.747512 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.747595 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.747600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.748278 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.748281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.748806 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.748814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.749241 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.749245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.749333 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.749768 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.749779 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.750059 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.751848 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.751851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.751853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.752089 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.752387 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.752390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.752392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.752478 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.752891 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.752894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.752990 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.753094 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.753244 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:56:55.776911 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:39051
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:55.885847 11073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:55.886106 11073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:55.886148 11073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:55.890021 11073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:55.890072 11073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:55.890115 11073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:55.890199 11073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:56:55.895643 11073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:39051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:55.896584 11073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:55.897555 11073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:55.906838 11073 server_base.cc:1034] running on GCE node
W20260806 23:56:55.906800 11078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:55.906785 11079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:55.906808 11081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:55.907367 11073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:55.907954 11073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:55.909147 11073 hybrid_clock.cc:648] HybridClock initialized: now 1786060615909135 us; error 24 us; skew 500 ppm
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:55.911983 11073 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:56:55.912794 11073 webserver.cc:460] Webserver started at http://127.8.204.67:38625/ using document root <none> and password file <none>
I20260806 23:56:55.913371 11073 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:55.913465 11073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:55.913704 11073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:55.915827 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "4fdc73becfa548729af8332dea9bb0c5"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "292aaae993fd28acb4736a9f062a4509"
server_key_iv: "523a92bb3d022916495addf4168614a7"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.916361 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "4fdc73becfa548729af8332dea9bb0c5"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "292aaae993fd28acb4736a9f062a4509"
server_key_iv: "523a92bb3d022916495addf4168614a7"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.919798 11073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 23:56:55.922158 11088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.923355 11073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:55.923499 11073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "4fdc73becfa548729af8332dea9bb0c5"
format_stamp: "Formatted at 2026-08-06 23:56:55 on dist-test-slave-39l8"
server_key: "292aaae993fd28acb4736a9f062a4509"
server_key_iv: "523a92bb3d022916495addf4168614a7"
server_key_version: "encryptionkey@0"
I20260806 23:56:55.923611 11073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:55.937906 11073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:55.941027 11073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:55.941226 11073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:55.942550 11073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:55.942631 11073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:55.942696 11073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:55.942790 11073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:55.953994 11073 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:45671
I20260806 23:56:55.954064 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:45671 every 8 connection(s)
I20260806 23:56:55.955092 11073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:55.956771 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.943413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:38651 (local address 127.8.204.126:39051)
0806 23:56:55.943562 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.943567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.944394 (+ 827us) server_negotiation.cc:407] Connection header received
0806 23:56:55.945543 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.945549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.945619 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.945728 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.947014 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.947660 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.948956 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.949194 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.949307 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.952681 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.952686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.952689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.952724 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.954422 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.954993 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.954999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.955004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.955114 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.955574 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.955580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.955581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.955827 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.955943 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.956480 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.956603 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:56:55.957445 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.943765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.67:38651)
0806 23:56:55.944231 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.944279 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.945283 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.945902 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.945910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.946268 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.946866 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.946882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.947789 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.947793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.948830 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.948840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.949346 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.949350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.949529 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.950153 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.950178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.952474 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.954551 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.954556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.954566 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.954888 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.955259 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.955262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.955264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.955398 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.955953 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.955961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.956200 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.956417 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.956723 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260806 23:56:55.958659 11259 heartbeater.cc:347] Connected to a master server at 127.8.204.126:39051
I20260806 23:56:55.958815 11259 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:55.959348 11259 heartbeater.cc:511] Master 127.8.204.126:39051 requested a full tablet report, sending...
I20260806 23:56:55.960436 10595 ts_manager.cc:194] Registered new tserver with Master: 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:55.961184 10595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:38651
I20260806 23:56:55.962962 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 11073
I20260806 23:56:55.963114 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:55.963383 9009 external_mini_cluster.cc:1428] Setting key 030080c3b9d702869e5940b52c006f23
I20260806 23:56:55.965523 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:55.968650 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.960878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.8.204.67:41179)
0806 23:56:55.961564 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:55.961587 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:55.961704 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:55.961953 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:55.961956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:55.962127 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:55.962227 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.962232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.963288 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.963291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.963917 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:55.963928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.964484 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.964487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:55.964555 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.965048 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:55.965059 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:55.965419 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:55.967155 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.967161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.967166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.967457 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.967716 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:55.967721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:55.967726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:55.967827 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:55.968289 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:55.968291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:55.968363 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 23:56:55.968444 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 23:56:55.968537 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":80,"threads_started":1}
I20260806 23:56:55.968837 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.960946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:41179 (local address 127.8.204.126:39051)
0806 23:56:55.961053 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.961056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.961662 (+ 606us) server_negotiation.cc:407] Connection header received
0806 23:56:55.961799 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.961802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.961846 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.961930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.962342 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.963092 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.964043 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.964360 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.964470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.965565 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.965568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.965570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.965620 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.967035 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.967558 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.967560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.967561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.967631 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.967941 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.967947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.967948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.968188 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.968292 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.968529 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.968651 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:56:55 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:55.996568 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:55.988391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.8.204.126:39051)
0806 23:56:55.988534 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:55.988538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:55.988550 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:56:55.988586 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:55.988588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:55.988623 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:55.988711 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:55.989370 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.989900 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.990745 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:55.990963 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:55.991069 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:55.993470 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:55.993479 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:55.993480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:55.993504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:55.994922 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.995364 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.995367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.995368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.995437 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:55.995744 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:55.995747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:55.995749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:55.995915 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:55.996049 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:55.996297 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 23:56:55.996410 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:55.999521 10595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:56.002413 10595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:56.017234 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.010667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44103 (local address 127.0.0.1:58656)
0806 23:56:56.011437 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.011451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.011543 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.012304 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.012309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.012332 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.012515 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.012522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.013983 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.013988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.015485 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.015498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.016708 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.016715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.016837 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.016878 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.016979 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.017091 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":3,"thread_start_us":138,"threads_started":1}
I20260806 23:56:56.017947 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.011116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.8.204.65:44103)
0806 23:56:56.011509 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.011525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.011549 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:56:56.011724 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.011729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.011947 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.012136 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.012691 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.013861 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.015628 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.016386 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.016484 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.016646 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.017690 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.017782 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":93,"threads_started":1}
I20260806 23:56:56.018426 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.011126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45671 (local address 127.0.0.1:38608)
0806 23:56:56.013162 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.013174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.013254 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.013861 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.013864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.013884 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.014151 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.014162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.015563 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.015565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.016649 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.016661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.017977 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.017982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.018137 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.018165 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.018239 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.018305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1975,"mutex_wait_us":157,"thread_start_us":80,"threads_started":1}
I20260806 23:56:56.018841 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.011360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.8.204.67:45671)
0806 23:56:56.011889 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.011899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.013346 (+ 1447us) server_negotiation.cc:407] Connection header received
0806 23:56:56.013428 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.013433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.013600 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.013728 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.014290 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.015430 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.016838 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.017648 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.017740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.017873 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.018581 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.018671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":118,"threads_started":1}
I20260806 23:56:56.019672 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.010515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45009 (local address 127.0.0.1:47886)
0806 23:56:56.011323 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.011370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.011499 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.012822 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.012825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.012843 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.012946 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.012952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.016387 (+ 3435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.016390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.017477 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.017489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.019296 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.019300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.019385 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.019414 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.019497 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.019558 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":87,"threads_started":1}
I20260806 23:56:56.020042 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.010639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.8.204.66:45009)
0806 23:56:56.012146 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.012158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.012179 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:56:56.012252 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.012259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.012462 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.012670 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.013092 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.016228 (+ 3136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.017633 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.018425 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.018524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.019768 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.019821 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.019905 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":190,"threads_started":1}
I20260806 23:56:56.021128 10784 tablet_service.cc:1467] Processing CreateTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb (DEFAULT_TABLE table=test-table [id=5fa882d88ea145edb4684096f43b3f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.021482 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb (DEFAULT_TABLE table=test-table [id=5fa882d88ea145edb4684096f43b3f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.022341 10784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e136f011f9c48e3bcb7eeefa79917bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.022470 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e136f011f9c48e3bcb7eeefa79917bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.023267 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb (DEFAULT_TABLE table=test-table [id=5fa882d88ea145edb4684096f43b3f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.024160 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e136f011f9c48e3bcb7eeefa79917bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.029126 11291 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Bootstrap starting.
I20260806 23:56:56.030781 11292 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Bootstrap starting.
I20260806 23:56:56.031459 11291 tablet_bootstrap.cc:654] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.032429 11291 log.cc:824] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:56.032486 11293 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Bootstrap starting.
I20260806 23:56:56.032931 11292 tablet_bootstrap.cc:654] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.033656 11292 log.cc:824] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:56.034685 11293 tablet_bootstrap.cc:654] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.034739 11291 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: No bootstrap required, opened a new log
I20260806 23:56:56.034926 11291 ts_tablet_manager.cc:1397] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:56.035580 11292 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: No bootstrap required, opened a new log
I20260806 23:56:56.035557 11293 log.cc:824] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:56.035773 11292 ts_tablet_manager.cc:1397] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:56:56.037503 11293 tablet_bootstrap.cc:492] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: No bootstrap required, opened a new log
I20260806 23:56:56.037744 11293 ts_tablet_manager.cc:1397] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260806 23:56:56.038857 11291 raft_consensus.cc:348] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.039127 11291 raft_consensus.cc:374] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.038977 11292 raft_consensus.cc:348] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.039202 11292 raft_consensus.cc:374] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.039218 11291 raft_consensus.cc:729] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdc73becfa548729af8332dea9bb0c5, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.039278 11292 raft_consensus.cc:729] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7009d7aed7714e2dbb34fb7220152a5f, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.039716 11292 consensus_queue.cc:260] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.039753 11291 consensus_queue.cc:260] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.040300 10873 heartbeater.cc:503] Master 127.8.204.126:39051 was elected leader, sending a full tablet report...
I20260806 23:56:56.040673 11292 ts_tablet_manager.cc:1428] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:56.040822 11291 ts_tablet_manager.cc:1428] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:56:56.041227 11259 heartbeater.cc:503] Master 127.8.204.126:39051 was elected leader, sending a full tablet report...
I20260806 23:56:56.041527 11293 raft_consensus.cc:348] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.041718 11293 raft_consensus.cc:374] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.041749 11293 raft_consensus.cc:729] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b079c298454454bd05c1549234025d, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.042071 11293 consensus_queue.cc:260] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.042760 11064 heartbeater.cc:503] Master 127.8.204.126:39051 was elected leader, sending a full tablet report...
I20260806 23:56:56.043048 11293 ts_tablet_manager.cc:1428] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:56.050889 11298 raft_consensus.cc:482] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:56.050998 11298 raft_consensus.cc:504] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.051826 11298 leader_election.cc:288] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b079c298454454bd05c1549234025d (127.8.204.66:45009), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.056814 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.052169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:45159 (local address 127.8.204.66:45009)
0806 23:56:56.052278 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.052283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.052417 (+ 134us) server_negotiation.cc:407] Connection header received
0806 23:56:56.052589 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.052592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.052639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.052720 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.053052 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.053847 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.055640 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.056197 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.056312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.056360 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.056640 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.056702 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:56.056814 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.052144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:58521 (local address 127.8.204.67:45671)
0806 23:56:56.052263 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.052268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.052494 (+ 226us) server_negotiation.cc:407] Connection header received
0806 23:56:56.052615 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.052620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.052660 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.052748 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.053111 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.053847 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.055640 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.056197 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.056312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.056360 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.056640 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.056702 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:56:56.056854 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.052065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45671 (local address 127.8.204.65:58521)
0806 23:56:56.052375 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.052386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.052533 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.052760 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.052763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.052806 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.052954 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.052962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.053983 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.053985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.055520 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.055531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.056334 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.056338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.056493 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.056536 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.056610 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.056701 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":237,"spinlock_wait_cycles":3712,"thread_start_us":53,"threads_started":1}
I20260806 23:56:56.056856 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.052028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45009 (local address 127.8.204.65:45159)
0806 23:56:56.052333 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.052344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.052482 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.052731 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.052734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.052750 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.052926 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.052937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.053981 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.053985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.055520 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.055531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.056334 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.056338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.056495 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.056537 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.056610 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.056701 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":218,"spinlock_wait_cycles":20352,"thread_start_us":91,"threads_started":1}
I20260806 23:56:56.057374 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e136f011f9c48e3bcb7eeefa79917bb" candidate_uuid: "7009d7aed7714e2dbb34fb7220152a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5" is_pre_election: true
I20260806 23:56:56.057657 11195 raft_consensus.cc:2454] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7009d7aed7714e2dbb34fb7220152a5f in term 0.
I20260806 23:56:56.057780 10995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e136f011f9c48e3bcb7eeefa79917bb" candidate_uuid: "7009d7aed7714e2dbb34fb7220152a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b079c298454454bd05c1549234025d" is_pre_election: true
I20260806 23:56:56.058012 10995 raft_consensus.cc:2454] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7009d7aed7714e2dbb34fb7220152a5f in term 0.
I20260806 23:56:56.058090 10720 leader_election.cc:302] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdc73becfa548729af8332dea9bb0c5, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.058410 11298 raft_consensus.cc:2781] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:56.058478 11298 raft_consensus.cc:482] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:56.058506 11298 raft_consensus.cc:3037] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.059504 11298 raft_consensus.cc:504] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.059975 11298 leader_election.cc:288] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [CANDIDATE]: Term 1 election: Requested vote from peers 51b079c298454454bd05c1549234025d (127.8.204.66:45009), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.060290 10995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e136f011f9c48e3bcb7eeefa79917bb" candidate_uuid: "7009d7aed7714e2dbb34fb7220152a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b079c298454454bd05c1549234025d"
I20260806 23:56:56.060294 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e136f011f9c48e3bcb7eeefa79917bb" candidate_uuid: "7009d7aed7714e2dbb34fb7220152a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5"
I20260806 23:56:56.060415 10995 raft_consensus.cc:3037] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.060422 11195 raft_consensus.cc:3037] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.061609 11195 raft_consensus.cc:2454] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7009d7aed7714e2dbb34fb7220152a5f in term 1.
I20260806 23:56:56.061637 10995 raft_consensus.cc:2454] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7009d7aed7714e2dbb34fb7220152a5f in term 1.
I20260806 23:56:56.061899 10720 leader_election.cc:302] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdc73becfa548729af8332dea9bb0c5, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.062100 11298 raft_consensus.cc:2781] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:56.062305 11298 raft_consensus.cc:686] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 1 LEADER]: Becoming Leader. State: Replica: 7009d7aed7714e2dbb34fb7220152a5f, State: Running, Role: LEADER
I20260806 23:56:56.062603 11298 consensus_queue.cc:237] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.065485 10595 catalog_manager.cc:5495] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65). New cstate: current_term: 1 leader_uuid: "7009d7aed7714e2dbb34fb7220152a5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:56.088835 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.085273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.8.204.65:44103)
0806 23:56:56.085387 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.085391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.085403 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:56:56.085590 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.085592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.085674 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.085785 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:56.086026 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.086568 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.087604 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.087841 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.087921 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.088193 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.088540 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.088632 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.088694 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.088738 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:56.094386 10995 raft_consensus.cc:1261] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Refusing update from remote peer 7009d7aed7714e2dbb34fb7220152a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.094424 11195 raft_consensus.cc:1261] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Refusing update from remote peer 7009d7aed7714e2dbb34fb7220152a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.095034 11298 consensus_queue.cc:1035] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.095237 11305 consensus_queue.cc:1035] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.104204 11315 mvcc.cc:205] Tried to move back new op lower bound from 7315704283515691008 to 7315704283395301376. Current Snapshot: MvccSnapshot[applied={T|T < 7315704283515691008}]
I20260806 23:56:56.112670 10595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:56.112915 10595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:56.115270 10595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Sending DeleteTablet for 3 replicas of tablet 9e136f011f9c48e3bcb7eeefa79917bb
I20260806 23:56:56.116150 10971 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:47886
I20260806 23:56:56.116361 10784 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:58656
I20260806 23:56:56.116354 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e136f011f9c48e3bcb7eeefa79917bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 23:56:56.116851 11324 tablet_replica.cc:330] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: stopping tablet replica
I20260806 23:56:56.116940 10595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:56:56.117163 11324 raft_consensus.cc:2229] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 23:56:56.117326 10595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:56.117516 11324 raft_consensus.cc:2258] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.118860 11325 tablet_replica.cc:330] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: stopping tablet replica
I20260806 23:56:56.119123 11325 raft_consensus.cc:2229] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:56.119994 11324 ts_tablet_manager.cc:1905] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.117944 11327 tablet_replica.cc:330] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: stopping tablet replica
I20260806 23:56:56.120636 11327 raft_consensus.cc:2229] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:56.120923 11327 raft_consensus.cc:2258] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.121206 11325 raft_consensus.cc:2258] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.121745 10784 tablet_service.cc:1467] Processing CreateTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 (DEFAULT_TABLE table=test-table [id=5f15a0abc3b8499ab8e278221a36113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.121919 10971 tablet_service.cc:1467] Processing CreateTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 (DEFAULT_TABLE table=test-table [id=5f15a0abc3b8499ab8e278221a36113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.122018 10784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddea9024d9e7478aa7c4e35053adcee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.122164 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddea9024d9e7478aa7c4e35053adcee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.123272 11327 ts_tablet_manager.cc:1905] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.124090 11292 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Bootstrap starting.
I20260806 23:56:56.124490 11293 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Bootstrap starting.
I20260806 23:56:56.125047 11292 tablet_bootstrap.cc:654] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.125393 11293 tablet_bootstrap.cc:654] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.125479 11325 ts_tablet_manager.cc:1905] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.126339 11165 tablet_service.cc:1467] Processing CreateTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 (DEFAULT_TABLE table=test-table [id=5f15a0abc3b8499ab8e278221a36113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:56.126511 11293 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: No bootstrap required, opened a new log
I20260806 23:56:56.126597 11293 ts_tablet_manager.cc:1397] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:56.126606 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddea9024d9e7478aa7c4e35053adcee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.127053 11293 raft_consensus.cc:348] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.127178 11293 raft_consensus.cc:374] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.127206 11293 raft_consensus.cc:729] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b079c298454454bd05c1549234025d, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.127355 11293 consensus_queue.cc:260] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.127605 11293 ts_tablet_manager.cc:1428] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:56:56.128878 11291 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Bootstrap starting.
I20260806 23:56:56.129177 11327 ts_tablet_manager.cc:1918] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:56.129273 11327 log.cc:1196] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/9e136f011f9c48e3bcb7eeefa79917bb
I20260806 23:56:56.129596 11327 ts_tablet_manager.cc:1939] T 9e136f011f9c48e3bcb7eeefa79917bb P 4fdc73becfa548729af8332dea9bb0c5: Deleting consensus metadata
I20260806 23:56:56.129887 11291 tablet_bootstrap.cc:654] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.131628 11325 ts_tablet_manager.cc:1918] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:56.131723 11325 log.cc:1196] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/9e136f011f9c48e3bcb7eeefa79917bb
I20260806 23:56:56.132041 11325 ts_tablet_manager.cc:1939] T 9e136f011f9c48e3bcb7eeefa79917bb P 7009d7aed7714e2dbb34fb7220152a5f: Deleting consensus metadata
I20260806 23:56:56.132530 11324 ts_tablet_manager.cc:1918] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:56.132653 11324 log.cc:1196] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/9e136f011f9c48e3bcb7eeefa79917bb
I20260806 23:56:56.132949 11324 ts_tablet_manager.cc:1939] T 9e136f011f9c48e3bcb7eeefa79917bb P 51b079c298454454bd05c1549234025d: Deleting consensus metadata
I20260806 23:56:56.133149 10576 catalog_manager.cc:4841] TS 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103): tablet 9e136f011f9c48e3bcb7eeefa79917bb (table test-table [id=5fa882d88ea145edb4684096f43b3f8d]) successfully deleted
I20260806 23:56:56.133807 10578 catalog_manager.cc:4841] TS 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671): tablet 9e136f011f9c48e3bcb7eeefa79917bb (table test-table [id=5fa882d88ea145edb4684096f43b3f8d]) successfully deleted
I20260806 23:56:56.134322 10579 catalog_manager.cc:4841] TS 51b079c298454454bd05c1549234025d (127.8.204.66:45009): tablet 9e136f011f9c48e3bcb7eeefa79917bb (table test-table [id=5fa882d88ea145edb4684096f43b3f8d]) successfully deleted
I20260806 23:56:56.135200 11292 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: No bootstrap required, opened a new log
I20260806 23:56:56.135239 11291 tablet_bootstrap.cc:492] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: No bootstrap required, opened a new log
I20260806 23:56:56.135313 11291 ts_tablet_manager.cc:1397] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 23:56:56.135317 11292 ts_tablet_manager.cc:1397] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260806 23:56:56.135716 11292 raft_consensus.cc:348] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.135754 11291 raft_consensus.cc:348] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.135840 11292 raft_consensus.cc:374] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.135860 11291 raft_consensus.cc:374] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.135879 11292 raft_consensus.cc:729] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7009d7aed7714e2dbb34fb7220152a5f, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.135887 11291 raft_consensus.cc:729] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdc73becfa548729af8332dea9bb0c5, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.136014 11291 consensus_queue.cc:260] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.136031 11292 consensus_queue.cc:260] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.136328 11292 ts_tablet_manager.cc:1428] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:56:56.136899 11291 ts_tablet_manager.cc:1428] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:56:56.155144 11302 raft_consensus.cc:482] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:56.155249 11302 raft_consensus.cc:504] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.156270 11302 leader_election.cc:288] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.160907 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.156466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44103 (local address 127.8.204.66:40219)
0806 23:56:56.156699 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.156714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.156794 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.157230 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.157233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.157250 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.157382 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.157387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.158152 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.158157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.159599 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.159611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.160456 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.160460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.160577 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.160605 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.160698 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.160773 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":194,"spinlock_wait_cycles":34688}
I20260806 23:56:56.160949 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.156829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:40219 (local address 127.8.204.65:44103)
0806 23:56:56.156965 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.156969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.156980 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:56:56.157019 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.157022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.157059 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.157118 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.157536 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.158032 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.159740 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.160330 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.160467 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.160523 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.160739 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.160814 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:56:56.161109 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.156585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:39387 (local address 127.8.204.67:45671)
0806 23:56:56.156702 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.156707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.156983 (+ 276us) server_negotiation.cc:407] Connection header received
0806 23:56:56.157125 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.157128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.157173 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.157250 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:56.157571 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.158302 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.159817 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.160411 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.160486 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.160525 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.160909 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.160985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:56:56.161478 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddea9024d9e7478aa7c4e35053adcee5" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7009d7aed7714e2dbb34fb7220152a5f" is_pre_election: true
I20260806 23:56:56.161755 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.156493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45671 (local address 127.8.204.66:39387)
0806 23:56:56.156865 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.156879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.157019 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.157265 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.157267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.157296 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:56.157434 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.157439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.158410 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.158413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.159690 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.159699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.160638 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.160653 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.160772 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.160793 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.161542 (+ 749us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.161622 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":31,"thread_start_us":103,"threads_started":1}
I20260806 23:56:56.161721 10811 raft_consensus.cc:2454] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 0.
I20260806 23:56:56.162122 10906 leader_election.cc:302] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b079c298454454bd05c1549234025d, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.162281 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddea9024d9e7478aa7c4e35053adcee5" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5" is_pre_election: true
I20260806 23:56:56.162417 11195 raft_consensus.cc:2454] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 0.
I20260806 23:56:56.162632 11302 raft_consensus.cc:2781] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:56.162694 11302 raft_consensus.cc:482] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:56.162771 11302 raft_consensus.cc:3037] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.163718 11302 raft_consensus.cc:504] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.164088 11302 leader_election.cc:288] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 election: Requested vote from peers 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.164387 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddea9024d9e7478aa7c4e35053adcee5" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7009d7aed7714e2dbb34fb7220152a5f"
I20260806 23:56:56.164386 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddea9024d9e7478aa7c4e35053adcee5" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5"
I20260806 23:56:56.164494 10811 raft_consensus.cc:3037] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.164494 11195 raft_consensus.cc:3037] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.165601 10811 raft_consensus.cc:2454] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 1.
I20260806 23:56:56.165598 11195 raft_consensus.cc:2454] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 1.
I20260806 23:56:56.165877 10906 leader_election.cc:302] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b079c298454454bd05c1549234025d, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.166149 11302 raft_consensus.cc:2781] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:56.166407 11302 raft_consensus.cc:686] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 1 LEADER]: Becoming Leader. State: Replica: 51b079c298454454bd05c1549234025d, State: Running, Role: LEADER
I20260806 23:56:56.166651 11302 consensus_queue.cc:237] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.169050 10595 catalog_manager.cc:5495] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b079c298454454bd05c1549234025d (127.8.204.66). New cstate: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
W20260806 23:56:56.206146 11260 tablet.cc:2369] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 23:56:56 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:56.225315 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.216310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:39051 (local address 127.0.0.1:39898)
0806 23:56:56.216563 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.216587 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.216705 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.216972 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.216974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.217339 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:56.217430 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.217435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.218475 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.218478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.219072 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.219082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.219730 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.219733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.219814 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.220404 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:56.220416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:56.221845 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:56.223824 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:56.223833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:56.223838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:56.224055 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:56.224404 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:56.224406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:56.224408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:56.224482 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:56.224891 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:56.224896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:56.224988 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.225073 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.225184 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260806 23:56:56.225488 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.216353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.8.204.126:39051)
0806 23:56:56.216488 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.216493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.216670 (+ 177us) server_negotiation.cc:407] Connection header received
0806 23:56:56.216787 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.216791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.216841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.216933 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:56.217546 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.218370 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.219213 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.219612 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.219715 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.221987 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:56.221992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:56.221994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:56.222021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:56.223705 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.224200 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.224203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.224205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.224298 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.224570 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.224573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.224575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.224788 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:56.224887 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.225158 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.225289 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":46}
I20260806 23:56:56.229562 10595 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:39898:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:56:56.230376 10595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:56.235613 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 2b650c80d2f4445381064f82d18a3c11 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad2ad261ad4b4a7fb192bdb36d334c6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:56.235694 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 2b650c80d2f4445381064f82d18a3c11 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad2ad261ad4b4a7fb192bdb36d334c6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:56.235864 10784 tablet_service.cc:1467] Processing CreateTablet for tablet 2b650c80d2f4445381064f82d18a3c11 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad2ad261ad4b4a7fb192bdb36d334c6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:56.236083 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b650c80d2f4445381064f82d18a3c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.236090 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b650c80d2f4445381064f82d18a3c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.236268 10784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b650c80d2f4445381064f82d18a3c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:56.238909 11291 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: Bootstrap starting.
I20260806 23:56:56.239053 11293 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: Bootstrap starting.
I20260806 23:56:56.239828 11293 tablet_bootstrap.cc:654] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.239944 11291 tablet_bootstrap.cc:654] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.240882 11292 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: Bootstrap starting.
I20260806 23:56:56.241151 11293 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: No bootstrap required, opened a new log
I20260806 23:56:56.241256 11293 ts_tablet_manager.cc:1397] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
W20260806 23:56:56.241664 11065 tablet.cc:2369] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:56.241927 11292 tablet_bootstrap.cc:654] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:56.241986 11293 raft_consensus.cc:348] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.242115 11293 raft_consensus.cc:374] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.242162 11293 raft_consensus.cc:729] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b079c298454454bd05c1549234025d, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.242303 11293 consensus_queue.cc:260] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.242605 11302 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.243218 11302 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.243297 11292 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: No bootstrap required, opened a new log
I20260806 23:56:56.243387 11292 ts_tablet_manager.cc:1397] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:56.243834 11292 raft_consensus.cc:348] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.243957 11292 raft_consensus.cc:374] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.244022 11292 raft_consensus.cc:729] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7009d7aed7714e2dbb34fb7220152a5f, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.244170 11292 consensus_queue.cc:260] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.244426 11293 ts_tablet_manager.cc:1428] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 23:56:56.244549 11298 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.244678 11298 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.246157 11292 ts_tablet_manager.cc:1428] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:56.246773 11291 tablet_bootstrap.cc:492] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: No bootstrap required, opened a new log
I20260806 23:56:56.246878 11291 ts_tablet_manager.cc:1397] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 23:56:56.247335 11291 raft_consensus.cc:348] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.247452 11291 raft_consensus.cc:374] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:56.247488 11291 raft_consensus.cc:729] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdc73becfa548729af8332dea9bb0c5, State: Initialized, Role: FOLLOWER
I20260806 23:56:56.247618 11291 consensus_queue.cc:260] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.248519 11299 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.248679 11299 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.249441 11291 ts_tablet_manager.cc:1428] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20260806 23:56:56.282080 10875 tablet.cc:2369] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:56.388500 11302 raft_consensus.cc:482] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:56.388655 11302 raft_consensus.cc:504] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.389159 11302 leader_election.cc:288] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.389676 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b650c80d2f4445381064f82d18a3c11" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5" is_pre_election: true
I20260806 23:56:56.389676 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b650c80d2f4445381064f82d18a3c11" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7009d7aed7714e2dbb34fb7220152a5f" is_pre_election: true
I20260806 23:56:56.389843 11195 raft_consensus.cc:2454] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 0.
I20260806 23:56:56.389863 10811 raft_consensus.cc:2454] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 0.
I20260806 23:56:56.390182 10906 leader_election.cc:302] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b079c298454454bd05c1549234025d, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.390385 11302 raft_consensus.cc:2781] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:56.390446 11302 raft_consensus.cc:482] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:56.390473 11302 raft_consensus.cc:3037] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.391520 11302 raft_consensus.cc:504] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.391882 11302 leader_election.cc:288] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 election: Requested vote from peers 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103), 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671)
I20260806 23:56:56.392282 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b650c80d2f4445381064f82d18a3c11" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7009d7aed7714e2dbb34fb7220152a5f"
I20260806 23:56:56.392303 11195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b650c80d2f4445381064f82d18a3c11" candidate_uuid: "51b079c298454454bd05c1549234025d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fdc73becfa548729af8332dea9bb0c5"
I20260806 23:56:56.392382 10811 raft_consensus.cc:3037] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.392421 11195 raft_consensus.cc:3037] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:56.393307 10811 raft_consensus.cc:2454] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 1.
I20260806 23:56:56.393313 11195 raft_consensus.cc:2454] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b079c298454454bd05c1549234025d in term 1.
I20260806 23:56:56.393581 10906 leader_election.cc:302] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b079c298454454bd05c1549234025d, 7009d7aed7714e2dbb34fb7220152a5f; no voters:
I20260806 23:56:56.393787 11302 raft_consensus.cc:2781] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:56.393895 11302 raft_consensus.cc:686] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [term 1 LEADER]: Becoming Leader. State: Replica: 51b079c298454454bd05c1549234025d, State: Running, Role: LEADER
I20260806 23:56:56.394029 11302 consensus_queue.cc:237] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } }
I20260806 23:56:56.394737 11332 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: TxnStatusTablet state changed. Reason: New leader 51b079c298454454bd05c1549234025d. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.394845 11332 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:56.395099 11351 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:56:56.395349 11351 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:56:56.395855 10595 catalog_manager.cc:5495] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b079c298454454bd05c1549234025d (127.8.204.66). New cstate: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:56.444237 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.440355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.8.204.66:45009)
0806 23:56:56.440487 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.440492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.440507 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:56:56.440679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.440683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.440758 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.440864 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:56.441075 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.441840 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.442613 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.442973 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.443071 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.443340 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.443862 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.443969 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.444027 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.444083 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:56:56.444240 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.440257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45009 (local address 127.0.0.1:47892)
0806 23:56:56.440400 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.440418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.440518 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.440879 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.440881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.440887 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:56.440976 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.440980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.441937 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.441939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.442509 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.442516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.443073 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.443075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.443151 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.443195 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.443952 (+ 757us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.443964 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.444013 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.444138 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 23:56:56.447093 11195 raft_consensus.cc:1261] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Refusing update from remote peer 51b079c298454454bd05c1549234025d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.447428 10811 raft_consensus.cc:1261] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Refusing update from remote peer 51b079c298454454bd05c1549234025d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.447611 11302 consensus_queue.cc:1035] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.447999 11332 consensus_queue.cc:1035] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.449620 11298 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: TxnStatusTablet state changed. Reason: New leader 51b079c298454454bd05c1549234025d. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.449744 11298 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.450114 11299 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: TxnStatusTablet state changed. Reason: New leader 51b079c298454454bd05c1549234025d. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.450258 11299 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.451647 11302 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.451756 11302 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:56.452564 11314 mvcc.cc:205] Tried to move back new op lower bound from 7315704284962340864 to 7315704284751421440. Current Snapshot: MvccSnapshot[applied={T|T < 7315704284962340864}]
I20260806 23:56:56.452755 11298 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.452850 11298 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 7009d7aed7714e2dbb34fb7220152a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.454612 11332 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.454770 11332 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 51b079c298454454bd05c1549234025d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:56.455534 11316 mvcc.cc:205] Tried to move back new op lower bound from 7315704284962340864 to 7315704284751421440. Current Snapshot: MvccSnapshot[applied={T|T < 7315704284962340864}]
I20260806 23:56:56.457202 11299 tablet_replica.cc:439] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51b079c298454454bd05c1549234025d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 } } peers { permanent_uuid: "51b079c298454454bd05c1549234025d" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45009 } } peers { permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 } } }
I20260806 23:56:56.457309 11299 tablet_replica.cc:442] T 2b650c80d2f4445381064f82d18a3c11 P 4fdc73becfa548729af8332dea9bb0c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:56.462469 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.459224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.8.204.66:45009)
0806 23:56:56.459323 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.459329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.459376 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:56:56.459542 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.459545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.459604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.459684 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:56.459957 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.460467 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.461288 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.461529 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.461646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.462017 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.462111 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.462197 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.462294 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.462344 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:56.472934 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.469809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:34299 (local address 127.8.204.66:45009)
0806 23:56:56.469930 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.469934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.469947 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:56.470065 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.470072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.470111 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.470186 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:56.470462 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.470983 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.471855 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.472077 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.472212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.472498 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.472557 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.472633 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.472723 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.472789 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":54}
I20260806 23:56:56.472949 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.469533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45009 (local address 127.8.204.66:34299)
0806 23:56:56.469826 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:56.469861 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:56.469966 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:56.470206 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:56.470209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:56.470216 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:56.470356 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.470362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.471083 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.471088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.471702 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:56.471711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.472226 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.472229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:56.472311 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.472358 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:56.472638 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:56.472646 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 23:56:56.472706 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 23:56:56.472797 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260806 23:56:56.479664 11195 raft_consensus.cc:1261] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Refusing update from remote peer 51b079c298454454bd05c1549234025d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.479743 10811 raft_consensus.cc:1261] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Refusing update from remote peer 51b079c298454454bd05c1549234025d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:56.480089 11359 consensus_queue.cc:1035] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdc73becfa548729af8332dea9bb0c5" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.480224 11332 consensus_queue.cc:1035] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7009d7aed7714e2dbb34fb7220152a5f" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:56.521740 10594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:56.521912 10594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:56.523958 10594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cbc80d57c421f826e42aab04d6da7: Sending DeleteTablet for 3 replicas of tablet ddea9024d9e7478aa7c4e35053adcee5
I20260806 23:56:56.524551 10784 tablet_service.cc:1514] Processing DeleteTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:58656
I20260806 23:56:56.524570 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 23:56:56.524748 11325 tablet_replica.cc:330] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: stopping tablet replica
I20260806 23:56:56.524783 10971 tablet_service.cc:1514] Processing DeleteTablet for tablet ddea9024d9e7478aa7c4e35053adcee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:47886
I20260806 23:56:56.524888 11325 raft_consensus.cc:2229] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:56.524928 11324 tablet_replica.cc:330] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: stopping tablet replica
I20260806 23:56:56.525009 11324 raft_consensus.cc:2229] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:56.525017 11325 raft_consensus.cc:2258] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.525183 11327 tablet_replica.cc:330] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: stopping tablet replica
I20260806 23:56:56.525305 11324 raft_consensus.cc:2258] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.525326 11327 raft_consensus.cc:2229] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:56.525429 11327 raft_consensus.cc:2258] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:56.526095 11324 ts_tablet_manager.cc:1905] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.526940 11327 ts_tablet_manager.cc:1905] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.529139 11324 ts_tablet_manager.cc:1918] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:56.529228 11324 log.cc:1196] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/ddea9024d9e7478aa7c4e35053adcee5
I20260806 23:56:56.529194 11327 ts_tablet_manager.cc:1918] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:56.529294 11327 log.cc:1196] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/ddea9024d9e7478aa7c4e35053adcee5
I20260806 23:56:56.529567 11327 ts_tablet_manager.cc:1939] T ddea9024d9e7478aa7c4e35053adcee5 P 4fdc73becfa548729af8332dea9bb0c5: Deleting consensus metadata
I20260806 23:56:56.529733 11324 ts_tablet_manager.cc:1939] T ddea9024d9e7478aa7c4e35053adcee5 P 51b079c298454454bd05c1549234025d: Deleting consensus metadata
I20260806 23:56:56.530342 10578 catalog_manager.cc:4841] TS 4fdc73becfa548729af8332dea9bb0c5 (127.8.204.67:45671): tablet ddea9024d9e7478aa7c4e35053adcee5 (table test-table [id=5f15a0abc3b8499ab8e278221a36113f]) successfully deleted
I20260806 23:56:56.530892 10579 catalog_manager.cc:4841] TS 51b079c298454454bd05c1549234025d (127.8.204.66:45009): tablet ddea9024d9e7478aa7c4e35053adcee5 (table test-table [id=5f15a0abc3b8499ab8e278221a36113f]) successfully deleted
I20260806 23:56:56.531064 11325 ts_tablet_manager.cc:1905] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:56.533641 11325 ts_tablet_manager.cc:1918] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:56.533717 11325 log.cc:1196] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/ddea9024d9e7478aa7c4e35053adcee5
I20260806 23:56:56.533946 11325 ts_tablet_manager.cc:1939] T ddea9024d9e7478aa7c4e35053adcee5 P 7009d7aed7714e2dbb34fb7220152a5f: Deleting consensus metadata
I20260806 23:56:56.534608 10576 catalog_manager.cc:4841] TS 7009d7aed7714e2dbb34fb7220152a5f (127.8.204.65:44103): tablet ddea9024d9e7478aa7c4e35053adcee5 (table test-table [id=5f15a0abc3b8499ab8e278221a36113f]) successfully deleted
Aug 06 23:56:56 dist-test-slave-39l8 krb5kdc[10535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:56:56.545593 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.526077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.8.204.65:44103)
0806 23:56:56.526220 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.526225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.530784 (+ 4559us) server_negotiation.cc:407] Connection header received
0806 23:56:56.534480 (+ 3696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.534482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.534530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.534608 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:56.535678 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.536476 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.537627 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.537949 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.538046 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.541774 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:56.541784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:56.541797 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:56.541822 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:56.543695 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.544178 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.544184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.544186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.544274 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.544563 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.544566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.544568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.544839 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:56.545034 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.545290 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.545406 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 23:56:56.545962 10845 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58674
I20260806 23:56:56.558684 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:56.547776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.8.204.126:39051)
0806 23:56:56.547880 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:56.547884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:56.551005 (+ 3121us) server_negotiation.cc:407] Connection header received
0806 23:56:56.551085 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:56.551087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:56.551134 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:56.551213 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:56.551870 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.552669 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.553703 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:56.554049 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:56.554157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:56.555443 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:56.555447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:56.555450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:56.555476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:56.556921 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.557328 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.557332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.557351 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.557437 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:56.557765 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:56.557768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:56.557770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:56.557972 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:56.558118 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:56.558391 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 23:56:56.558510 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 23:56:56.559324 10594 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39904
I20260806 23:56:56.560542 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 10694
I20260806 23:56:56.573046 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 10884
I20260806 23:56:56.584152 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 11073
I20260806 23:56:56.592710 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 10558
2026-08-06T23:56:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2787 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:56 dist-test-slave-39l8 krb5kdc[11376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:56 dist-test-slave-39l8 krb5kdc[11376](info): set up 2 sockets
Aug 06 23:56:56 dist-test-slave-39l8 krb5kdc[11376](info): commencing operation
krb5kdc: starting...
W20260806 23:56:57.760339 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:56:57 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:56:57.877758 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:42383
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:57.985323 11396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:57.985524 11396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:57.985558 11396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:57.989190 11396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:57.989230 11396 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:56:57.989246 11396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:57.989281 11396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:57.989297 11396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:57.989312 11396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:57.994181 11396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:42383
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:57.995018 11396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:57.995875 11396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:58.003561 11404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.003559 11406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.003611 11409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:58.003746 11396 server_base.cc:1034] running on GCE node
I20260806 23:56:58.004355 11396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:58.005321 11396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:58.006511 11396 hybrid_clock.cc:648] HybridClock initialized: now 1786060618006488 us; error 42 us; skew 500 ppm
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:58.014782 11396 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:56:58.015612 11396 webserver.cc:460] Webserver started at http://127.8.204.126:43815/ using document root <none> and password file <none>
I20260806 23:56:58.016132 11396 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:58.016224 11396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:58.016438 11396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:58.018908 11396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "71bce6c686344781b19b2a95f322b7d1"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "25320e0b512d44b251c3e24e2d92fe28"
server_key_iv: "18f344b02cd10df0ba75a80261ee4892"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.019575 11396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "71bce6c686344781b19b2a95f322b7d1"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "25320e0b512d44b251c3e24e2d92fe28"
server_key_iv: "18f344b02cd10df0ba75a80261ee4892"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.023746 11396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 23:56:58.025807 11419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.026953 11396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:58.027076 11396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "71bce6c686344781b19b2a95f322b7d1"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "25320e0b512d44b251c3e24e2d92fe28"
server_key_iv: "18f344b02cd10df0ba75a80261ee4892"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.027172 11396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:58.038612 11396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:58.041153 11396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:58.041337 11396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:58.048883 11396 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:42383
I20260806 23:56:58.048910 11521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:42383 every 8 connection(s)
I20260806 23:56:58.049741 11396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:58.052294 11522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.054073 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 11396
I20260806 23:56:58.054183 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:56:58.054442 9009 external_mini_cluster.cc:1428] Setting key 0f1824217b076e987be9c86407b8d402
I20260806 23:56:58.058228 11522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Bootstrap starting.
I20260806 23:56:58.060411 11522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.061164 11522 log.cc:824] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.062934 11522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: No bootstrap required, opened a new log
I20260806 23:56:58.065577 11522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } }
I20260806 23:56:58.065812 11522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.065894 11522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71bce6c686344781b19b2a95f322b7d1, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.066346 11522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } }
I20260806 23:56:58.066488 11522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:58.066581 11522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:58.066699 11522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.067557 11522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } }
I20260806 23:56:58.067660 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.055811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.8.204.126:42383)
0806 23:56:58.056197 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.056211 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.056242 (+ 31us) server_negotiation.cc:407] Connection header received
0806 23:56:58.056859 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.056873 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.057144 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.057445 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.058202 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.059225 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.060058 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.060529 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.060697 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.063041 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.063047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.063063 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.063091 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.065167 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.065592 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.065596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.065600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.065708 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.065973 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.065976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.065978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.066321 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.066522 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.066782 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.067035 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260806 23:56:58.067960 11522 leader_election.cc:302] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71bce6c686344781b19b2a95f322b7d1; no voters:
I20260806 23:56:58.068295 11522 leader_election.cc:288] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:58.068397 11534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:58.068667 11534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 71bce6c686344781b19b2a95f322b7d1, State: Running, Role: LEADER
I20260806 23:56:58.069002 11534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } }
I20260806 23:56:58.069509 11522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:58.070176 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71bce6c686344781b19b2a95f322b7d1. Latest consensus state: current_term: 1 leader_uuid: "71bce6c686344781b19b2a95f322b7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } } }
I20260806 23:56:58.070384 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:58.070678 11535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71bce6c686344781b19b2a95f322b7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71bce6c686344781b19b2a95f322b7d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42383 } } }
I20260806 23:56:58.070842 11543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:58.070899 11535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:58.073953 11543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:58.079015 11543 catalog_manager.cc:1347] Generated new cluster ID: ffbf3c1919e34f3d96891974ead7d86d
I20260806 23:56:58.079118 11543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:58.092000 11543 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:58.093209 11543 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:58.097290 11543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Generated new TSK 0
I20260806 23:56:58.097915 11543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:56:58.127408 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:58.239094 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:58.239344 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:58.239414 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:58.243366 11562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:58.243435 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:58.243501 11562 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:56:58.243543 11562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:58.243606 11562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:56:58.248450 11562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:58.249352 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:58.250313 11562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:58.259245 11572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.259243 11568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.259243 11567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:58.259367 11562 server_base.cc:1034] running on GCE node
I20260806 23:56:58.259858 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:58.260454 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:58.261610 11562 hybrid_clock.cc:648] HybridClock initialized: now 1786060618261592 us; error 29 us; skew 500 ppm
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:58.264273 11562 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:56:58.265066 11562 webserver.cc:460] Webserver started at http://127.8.204.65:42873/ using document root <none> and password file <none>
I20260806 23:56:58.265601 11562 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:58.265690 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:58.265918 11562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:58.267899 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "aa4c05eeb23b41888757b0ff22b603c6"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "aa34e7c65112508465dc0ed8f819debd"
server_key_iv: "d1d343211c83b9d293d280ade59cd097"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.268456 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "aa4c05eeb23b41888757b0ff22b603c6"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "aa34e7c65112508465dc0ed8f819debd"
server_key_iv: "d1d343211c83b9d293d280ade59cd097"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.272382 11562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 23:56:58.274596 11586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.275715 11562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:58.275848 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "aa4c05eeb23b41888757b0ff22b603c6"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "aa34e7c65112508465dc0ed8f819debd"
server_key_iv: "d1d343211c83b9d293d280ade59cd097"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.275954 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:58.310305 11562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:58.313042 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:58.313290 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:58.314606 11562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:58.314674 11562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.314759 11562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:58.314807 11562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.325457 11562 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:41271
I20260806 23:56:58.325730 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:41271 every 8 connection(s)
I20260806 23:56:58.326501 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:58.328425 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.315411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:45265 (local address 127.8.204.126:42383)
0806 23:56:58.315527 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.315530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.316518 (+ 988us) server_negotiation.cc:407] Connection header received
0806 23:56:58.317621 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.317624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.317685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.317743 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.319329 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.320043 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.321147 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.321542 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.321673 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.324721 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.324725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.324728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.324754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.326304 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.326889 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.326895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.326899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.326988 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.327403 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.327406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.327409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.327603 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.327721 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.328143 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.328251 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:56:58.329057 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.315900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.65:45265)
0806 23:56:58.316380 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.316429 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.317388 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.318146 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.318154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.318573 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.319179 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.319204 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.320175 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.320179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.321022 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.321032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.321716 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.321721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.321913 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.322469 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.322485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.324558 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.326418 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.326422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.326431 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.326778 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.327119 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.327122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.327123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.327283 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.327721 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.327727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.327931 (+ 204us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.328095 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.328376 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260806 23:56:58.330277 11746 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42383
I20260806 23:56:58.330439 11746 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:58.330981 11746 heartbeater.cc:511] Master 127.8.204.126:42383 requested a full tablet report, sending...
I20260806 23:56:58.332619 11448 ts_manager.cc:194] Registered new tserver with Master: aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271)
I20260806 23:56:58.333679 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 11562
I20260806 23:56:58.333801 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:58.333966 11448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:45265
I20260806 23:56:58.334065 9009 external_mini_cluster.cc:1428] Setting key 801ecdec7b387aae4ff624f2d233f497
I20260806 23:56:58.339663 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.332369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.65:55049)
0806 23:56:58.332685 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.332700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.332830 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.333052 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.333055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.333251 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.333358 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.333365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.334231 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.334234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.335020 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.335030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.335657 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.335660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.335735 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.336152 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.336162 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.336438 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.338259 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.338262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.338263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.338444 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.338771 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.338774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.338775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.338853 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.339272 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.339274 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.339343 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.339421 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.339513 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260806 23:56:58.340016 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.332437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:55049 (local address 127.8.204.126:42383)
0806 23:56:58.332540 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.332543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.332767 (+ 224us) server_negotiation.cc:407] Connection header received
0806 23:56:58.332909 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.332912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.332947 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.333010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.333480 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.334113 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.335154 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.335432 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.335504 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.336569 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.336572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.336574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.336596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.338149 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.338551 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.338555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.338557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.338644 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.338927 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.338933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.338937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.339174 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.339282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.339508 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.339662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:56:58.359805 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:58.465214 11756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:58.465435 11756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:58.465478 11756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:58.469326 11756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:58.469373 11756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:58.469410 11756 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:56:58.469424 11756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:58.469453 11756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:56:58.474642 11756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:58.475507 11756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:58.476397 11756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:58.485486 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.485503 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:58.485702 11756 server_base.cc:1034] running on GCE node
W20260806 23:56:58.485841 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:58.486239 11756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:58.486856 11756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:58.488034 11756 hybrid_clock.cc:648] HybridClock initialized: now 1786060618488018 us; error 26 us; skew 500 ppm
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:58.490631 11756 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:56:58.491408 11756 webserver.cc:460] Webserver started at http://127.8.204.66:46395/ using document root <none> and password file <none>
I20260806 23:56:58.491856 11756 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:58.491909 11756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:58.492062 11756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:58.493942 11756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "9a005f8104a343828ebe1edbb901f02b"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "a7019c496c950b29448f49ab816f7277"
server_key_iv: "01d2a2076babeb33fc733470e41aada7"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.494422 11756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "9a005f8104a343828ebe1edbb901f02b"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "a7019c496c950b29448f49ab816f7277"
server_key_iv: "01d2a2076babeb33fc733470e41aada7"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.497853 11756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:56:58.500219 11776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.501314 11756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:56:58.501405 11756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "9a005f8104a343828ebe1edbb901f02b"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "a7019c496c950b29448f49ab816f7277"
server_key_iv: "01d2a2076babeb33fc733470e41aada7"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.501477 11756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:58.513953 11756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:58.516906 11756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:58.517056 11756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:58.518316 11756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:58.518384 11756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.518447 11756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:58.518494 11756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.528970 11756 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44155
I20260806 23:56:58.529147 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44155 every 8 connection(s)
I20260806 23:56:58.529985 11756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:58.532456 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.519074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:60955 (local address 127.8.204.126:42383)
0806 23:56:58.519241 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.519246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.520552 (+ 1306us) server_negotiation.cc:407] Connection header received
0806 23:56:58.521895 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.521899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.521952 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.522007 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.523755 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.524268 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.525982 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.526200 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.526283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.528705 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.528709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.528710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.528732 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.530402 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.531085 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.531089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.531090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.531208 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.531555 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.531558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.531559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.531725 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.531801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.532252 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.532343 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:56:58.533180 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.519906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.66:60955)
0806 23:56:58.520377 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.520438 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.521615 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.522419 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.522428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.522954 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.523589 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.523611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.524372 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.524380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.525863 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.525874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.526367 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.526371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.526593 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.527164 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.527178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.528528 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.530573 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.530602 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.530615 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.530982 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.531319 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.531321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.531322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.531461 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.531815 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.531819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.532040 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.532181 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.532398 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260806 23:56:58.534444 11938 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42383
I20260806 23:56:58.534592 11938 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:58.535127 11938 heartbeater.cc:511] Master 127.8.204.126:42383 requested a full tablet report, sending...
I20260806 23:56:58.535300 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 11756
I20260806 23:56:58.535418 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:58.535740 9009 external_mini_cluster.cc:1428] Setting key 8d2bb66346bf21036ea56381ab45585d
I20260806 23:56:58.536213 11450 ts_manager.cc:194] Registered new tserver with Master: 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155)
I20260806 23:56:58.537000 11450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:60955
I20260806 23:56:58.544548 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.537283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:44247 (local address 127.8.204.126:42383)
0806 23:56:58.537396 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.537399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.537408 (+ 9us) server_negotiation.cc:407] Connection header received
0806 23:56:58.537434 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.537435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.537464 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.537548 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.538035 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.538774 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.539831 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.540030 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.540116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.541308 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.541310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.541312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.541333 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.542736 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.543291 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.543298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.543302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.543421 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.543768 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.543771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.543772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.543958 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.544035 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.544367 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.544449 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:56:58.544687 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.536559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.66:44247)
0806 23:56:58.536833 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.536847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.536924 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.537569 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.537572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.537770 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.537909 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.537917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.538892 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.538895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.539701 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.539715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.540146 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.540152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.540287 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.540718 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.540733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.541167 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.542868 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.542872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.542875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.543151 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.543554 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.543559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.543564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.543659 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.544061 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.544066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.544181 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.544317 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.544476 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":22,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:56:58.563236 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:58.672380 11953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:58.672674 11953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:58.672780 11953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:58.676630 11953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:58.676712 11953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:58.676775 11953 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:56:58.676824 11953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:56:58.676890 11953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:56:58.682018 11953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:58.683048 11953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:58.684021 11953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:58.693612 11965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.693614 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:58.693639 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:58.693662 11953 server_base.cc:1034] running on GCE node
I20260806 23:56:58.694281 11953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:58.694919 11953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:58.696076 11953 hybrid_clock.cc:648] HybridClock initialized: now 1786060618696045 us; error 42 us; skew 500 ppm
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:58.698846 11953 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:56:58.699676 11953 webserver.cc:460] Webserver started at http://127.8.204.67:46519/ using document root <none> and password file <none>
I20260806 23:56:58.700249 11953 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:58.700343 11953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:58.700575 11953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:58.702566 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "ece443fef4793799e98835dae3312b5c"
server_key_iv: "957e6bc62dfb770a68122d97b9c0ff6d"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.703169 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "ece443fef4793799e98835dae3312b5c"
server_key_iv: "957e6bc62dfb770a68122d97b9c0ff6d"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.706895 11953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:58.709093 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.710314 11953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:58.710448 11953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
format_stamp: "Formatted at 2026-08-06 23:56:58 on dist-test-slave-39l8"
server_key: "ece443fef4793799e98835dae3312b5c"
server_key_iv: "957e6bc62dfb770a68122d97b9c0ff6d"
server_key_version: "encryptionkey@0"
I20260806 23:56:58.710554 11953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:58.727140 11953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:58.730476 11953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:58.730672 11953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:58.732290 11953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:58.732367 11953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:58.732434 11953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:58.732487 11953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.742777 11953 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:37409
I20260806 23:56:58.742848 12145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:37409 every 8 connection(s)
I20260806 23:56:58.743613 11953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:58.746273 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.733100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:55443 (local address 127.8.204.126:42383)
0806 23:56:58.733237 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.733241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.734273 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 23:56:58.735460 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.735466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.735531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.735632 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.737340 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.738163 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.739837 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.740169 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.740293 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.743053 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.743056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.743058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.743081 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.744545 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.745041 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.745044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.745045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.745119 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.745417 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.745420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.745421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.745588 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.745660 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.746074 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.746162 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:56:58.747061 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.733612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.67:55443)
0806 23:56:58.734081 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.734151 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.735233 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.736096 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.736106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.736501 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.737170 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.737190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.738303 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.738307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.739691 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.739722 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.740336 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.740340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.740514 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.741131 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.741153 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.742864 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.744668 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.744672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.744682 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.744938 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.745204 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.745206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.745207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.745331 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.745674 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.745679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.745880 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.746013 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.746232 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260806 23:56:58.748292 12146 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42383
I20260806 23:56:58.748466 12146 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:58.749007 12146 heartbeater.cc:511] Master 127.8.204.126:42383 requested a full tablet report, sending...
I20260806 23:56:58.749203 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 11953
I20260806 23:56:58.749361 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:58.749946 9009 external_mini_cluster.cc:1428] Setting key c6ce69d4de531db3c3a21ff0c91b0176
I20260806 23:56:58.750134 11448 ts_manager.cc:194] Registered new tserver with Master: 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:56:58.750921 11448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:55443
I20260806 23:56:58.754101 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:58.758872 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.751862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.8.204.67:54527)
0806 23:56:58.752129 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.752144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.752248 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.752447 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.752449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.752614 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.752704 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.752708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.753543 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.753546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.754184 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.754193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.754613 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.754617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.754754 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.755205 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.755216 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.755491 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.757387 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.757390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.757391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.757584 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.757904 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.757906 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.757907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.757980 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.758415 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.758417 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.758487 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.758565 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.758671 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260806 23:56:58.759016 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.751658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:54527 (local address 127.8.204.126:42383)
0806 23:56:58.751753 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.751759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.752251 (+ 492us) server_negotiation.cc:407] Connection header received
0806 23:56:58.752302 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.752305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.752348 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.752432 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.752816 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.753420 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.754297 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.754501 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.754601 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.755628 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.755631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.755633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.755655 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.757241 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.757703 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.757708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.757713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.757806 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.758073 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.758085 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.758089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.758318 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.758427 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.758654 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.758798 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.781682 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.773854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.8.204.126:42383)
0806 23:56:58.773960 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.773966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.773987 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:56:58.774020 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.774022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.774054 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.774145 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.774778 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.775275 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.776039 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.776236 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.776308 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.778544 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.778547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.778549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.778571 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.779978 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.780430 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.780435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.780437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.780520 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.780812 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.780815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.780817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.781058 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.781195 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.781416 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.781520 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:58.784212 11448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:58.787056 11448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:58.800004 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.794724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.0.0.1:57208)
0806 23:56:58.795211 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.795255 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.795377 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.796012 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.796019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.796044 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:58.796194 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.796202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.797383 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.797386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.798187 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.798197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.799368 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.799371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.799485 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.799544 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.799741 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.799869 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":59,"threads_started":1}
I20260806 23:56:58.800030 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.794871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.8.204.65:41271)
0806 23:56:58.795119 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.795133 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.795352 (+ 219us) server_negotiation.cc:407] Connection header received
0806 23:56:58.795629 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.795636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.795844 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.796004 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:58.796301 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.797103 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.798367 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.799231 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.799355 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.799499 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.799781 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.799877 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260806 23:56:58.801299 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.795686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44155 (local address 127.0.0.1:60192)
0806 23:56:58.796421 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.796450 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.796523 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.797013 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.797016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.797035 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:58.797141 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.797150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.798476 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.798480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.799530 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.799539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.800878 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.800882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.800990 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.801012 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.801087 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.801163 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":40,"thread_start_us":81,"threads_started":1}
I20260806 23:56:58.801735 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.795365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:37409 (local address 127.0.0.1:41134)
0806 23:56:58.796224 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.796236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.796316 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.797220 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.797223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.797242 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:58.797330 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.797338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.798677 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.798680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.800088 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.800101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.801307 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.801311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.801404 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.801422 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.801484 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.801629 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":172,"thread_start_us":97,"threads_started":1}
I20260806 23:56:58.802223 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.795536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.8.204.67:37409)
0806 23:56:58.796762 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.796772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.796792 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:56:58.796866 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.796870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.797018 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.797238 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:58.797460 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.798537 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.800236 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.801022 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.801127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.801306 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.801963 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.802070 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":118,"threads_started":1}
I20260806 23:56:58.802304 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.796064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.8.204.66:44155)
0806 23:56:58.796567 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.796591 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.796617 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:56:58.796691 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.796696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.796874 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.797008 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:58.797278 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.798337 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.799663 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.800739 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.800864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.801016 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.801393 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.801490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":96,"threads_started":1}
I20260806 23:56:58.803270 11653 tablet_service.cc:1467] Processing CreateTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 (DEFAULT_TABLE table=test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:58.804425 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acae4742e694d5bb61dd8c24c5c56b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.804378 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 (DEFAULT_TABLE table=test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:58.804620 12044 tablet_service.cc:1467] Processing CreateTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 (DEFAULT_TABLE table=test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:58.805256 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acae4742e694d5bb61dd8c24c5c56b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.805497 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acae4742e694d5bb61dd8c24c5c56b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.812283 12175 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Bootstrap starting.
I20260806 23:56:58.812894 12176 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Bootstrap starting.
I20260806 23:56:58.814625 12175 tablet_bootstrap.cc:654] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.814832 12176 tablet_bootstrap.cc:654] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.815111 12177 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Bootstrap starting.
I20260806 23:56:58.815512 12176 log.cc:824] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:58.815512 12175 log.cc:824] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:58.816926 12177 tablet_bootstrap.cc:654] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.817420 12176 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: No bootstrap required, opened a new log
I20260806 23:56:58.817628 12176 ts_tablet_manager.cc:1397] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260806 23:56:58.817747 12177 log.cc:824] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:58.818867 12175 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: No bootstrap required, opened a new log
I20260806 23:56:58.819049 12175 ts_tablet_manager.cc:1397] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:56:58.819543 12177 tablet_bootstrap.cc:492] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: No bootstrap required, opened a new log
I20260806 23:56:58.819753 12177 ts_tablet_manager.cc:1397] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:56:58.821566 12176 raft_consensus.cc:348] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.821820 12176 raft_consensus.cc:374] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.821874 12176 raft_consensus.cc:729] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.822422 12176 consensus_queue.cc:260] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.822626 12177 raft_consensus.cc:348] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.822893 12177 raft_consensus.cc:374] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.822767 12175 raft_consensus.cc:348] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.822957 12177 raft_consensus.cc:729] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91071d4612bb49b69c06b6d4f8feb8a3, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.822997 12175 raft_consensus.cc:374] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.823055 12175 raft_consensus.cc:729] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.823387 11746 heartbeater.cc:503] Master 127.8.204.126:42383 was elected leader, sending a full tablet report...
I20260806 23:56:58.823488 12176 ts_tablet_manager.cc:1428] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:58.823436 12175 consensus_queue.cc:260] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.823457 12177 consensus_queue.cc:260] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.824111 12146 heartbeater.cc:503] Master 127.8.204.126:42383 was elected leader, sending a full tablet report...
I20260806 23:56:58.824326 12177 ts_tablet_manager.cc:1428] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:58.825067 12175 ts_tablet_manager.cc:1428] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:58.825393 11938 heartbeater.cc:503] Master 127.8.204.126:42383 was elected leader, sending a full tablet report...
W20260806 23:56:58.827584 11747 tablet.cc:2369] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:58.862579 12182 raft_consensus.cc:482] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:58.862757 12182 raft_consensus.cc:504] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.863643 12182 leader_election.cc:288] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409), 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155)
I20260806 23:56:58.867533 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.864012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:51925 (local address 127.8.204.67:37409)
0806 23:56:58.864184 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.864188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.864231 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:56:58.864400 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.864404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.864452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.864528 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:58.864889 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.865465 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.866616 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.867001 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.867109 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.867158 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.867351 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.867421 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:56:58.867551 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.864388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:49743 (local address 127.8.204.66:44155)
0806 23:56:58.864550 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.864554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.864586 (+ 32us) server_negotiation.cc:407] Connection header received
0806 23:56:58.864749 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.864752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.864803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.864879 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:58.865133 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.865646 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.866610 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.867016 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.867136 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.867180 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.867397 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.867463 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 23:56:58.867545 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.863875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:37409 (local address 127.8.204.65:51925)
0806 23:56:58.864144 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.864157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.864282 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.864567 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.864572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.864597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:58.864750 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.864757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.865576 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.865579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.866488 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.866498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.867111 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.867114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.867222 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.867244 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.867313 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.867394 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260806 23:56:58.867561 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.864238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44155 (local address 127.8.204.65:49743)
0806 23:56:58.864528 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.864541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.864647 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.864901 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.864903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.864922 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:58.865028 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.865033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.865760 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.865762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.866497 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.866505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.867148 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.867152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.867262 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.867279 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.867369 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.867428 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"spinlock_wait_cycles":2304,"thread_start_us":83,"threads_started":1}
I20260806 23:56:58.868099 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acae4742e694d5bb61dd8c24c5c56b9" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" is_pre_election: true
I20260806 23:56:58.868270 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acae4742e694d5bb61dd8c24c5c56b9" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b" is_pre_election: true
I20260806 23:56:58.868350 12077 raft_consensus.cc:2454] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:56:58.868512 11874 raft_consensus.cc:2454] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:56:58.868815 11592 leader_election.cc:302] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91071d4612bb49b69c06b6d4f8feb8a3, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:56:58.869016 12182 raft_consensus.cc:2781] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:58.869091 12182 raft_consensus.cc:482] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:58.869119 12182 raft_consensus.cc:3037] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.870281 12182 raft_consensus.cc:504] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.870625 12182 leader_election.cc:288] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Requested vote from peers 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409), 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155)
I20260806 23:56:58.871047 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acae4742e694d5bb61dd8c24c5c56b9" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b"
I20260806 23:56:58.871107 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acae4742e694d5bb61dd8c24c5c56b9" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
I20260806 23:56:58.871192 11874 raft_consensus.cc:3037] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.871254 12077 raft_consensus.cc:3037] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.872277 11874 raft_consensus.cc:2454] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:56:58.872465 12077 raft_consensus.cc:2454] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:56:58.872674 11592 leader_election.cc:302] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:56:58.872864 12182 raft_consensus.cc:2781] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:58.873061 12182 raft_consensus.cc:686] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 LEADER]: Becoming Leader. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Running, Role: LEADER
I20260806 23:56:58.873363 12182 consensus_queue.cc:237] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } }
I20260806 23:56:58.876149 11450 catalog_manager.cc:5495] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:56:58 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060618, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:56:58.916317 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.908155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.8.204.126:42383)
0806 23:56:58.908307 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:58.908312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:58.908452 (+ 140us) server_negotiation.cc:407] Connection header received
0806 23:56:58.908626 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:58.908629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:58.908672 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:58.908751 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:58.909400 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.909925 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.910684 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.910945 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.911050 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.913333 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:58.913336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:58.913341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:58.913365 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:58.914585 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.915042 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.915048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.915050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.915119 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:58.915383 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:58.915386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:58.915387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:58.915558 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:58.915659 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:58.915974 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 23:56:58.916153 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":73}
I20260806 23:56:58.916329 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:58.908097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42383 (local address 127.0.0.1:45564)
0806 23:56:58.908373 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:58.908384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:58.908526 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:58.908769 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:58.908773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:58.909153 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:58.909254 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.909261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.910032 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.910037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.910568 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:58.910575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:58.911061 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:58.911065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:58.911182 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:58.911791 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:58.911803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:58.913178 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:58.914696 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.914727 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.914733 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.914948 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.915210 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:58.915214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:58.915216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:58.915294 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:58.915670 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:58.915678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:58.915791 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 23:56:58.915928 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 23:56:58.916082 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":18,"thread_start_us":90,"threads_started":1}
I20260806 23:56:58.920076 11450 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:45564:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:56:58.920699 11450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:58.925729 12044 tablet_service.cc:1467] Processing CreateTablet for tablet d91062abe57b451bb29aea0a20d5177b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b65f9d90cd94ba491111f5162f8dd2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:58.925559 11653 tablet_service.cc:1467] Processing CreateTablet for tablet d91062abe57b451bb29aea0a20d5177b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b65f9d90cd94ba491111f5162f8dd2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:58.925745 11848 tablet_service.cc:1467] Processing CreateTablet for tablet d91062abe57b451bb29aea0a20d5177b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b65f9d90cd94ba491111f5162f8dd2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:56:58.926301 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91062abe57b451bb29aea0a20d5177b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.926301 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91062abe57b451bb29aea0a20d5177b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.926424 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91062abe57b451bb29aea0a20d5177b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:58.929242 12175 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: Bootstrap starting.
I20260806 23:56:58.929307 12176 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: Bootstrap starting.
I20260806 23:56:58.929644 12177 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: Bootstrap starting.
I20260806 23:56:58.930307 12175 tablet_bootstrap.cc:654] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.930796 12177 tablet_bootstrap.cc:654] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.930796 12176 tablet_bootstrap.cc:654] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:58.931833 12175 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: No bootstrap required, opened a new log
I20260806 23:56:58.931936 12175 ts_tablet_manager.cc:1397] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:58.932190 12176 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: No bootstrap required, opened a new log
I20260806 23:56:58.932294 12176 ts_tablet_manager.cc:1397] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 23:56:58.932461 12177 tablet_bootstrap.cc:492] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: No bootstrap required, opened a new log
I20260806 23:56:58.932435 12175 raft_consensus.cc:348] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.932542 12175 raft_consensus.cc:374] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.932551 12177 ts_tablet_manager.cc:1397] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 23:56:58.932590 12175 raft_consensus.cc:729] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.932734 12175 consensus_queue.cc:260] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.932999 12175 ts_tablet_manager.cc:1428] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:56:58.933017 12177 raft_consensus.cc:348] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.933020 12187 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:58.933118 12177 raft_consensus.cc:374] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.933171 12177 raft_consensus.cc:729] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91071d4612bb49b69c06b6d4f8feb8a3, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.933184 12187 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:58.933264 12177 consensus_queue.cc:260] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.933512 12177 ts_tablet_manager.cc:1428] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:56:58.933485 12186 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:58.933637 12186 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:58.932802 12176 raft_consensus.cc:348] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.934201 12176 raft_consensus.cc:374] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:58.934263 12176 raft_consensus.cc:729] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Initialized, Role: FOLLOWER
I20260806 23:56:58.934408 12176 consensus_queue.cc:260] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.934665 12182 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:58.934808 12182 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:58.935158 12176 ts_tablet_manager.cc:1428] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 23:56:58.988970 12182 raft_consensus.cc:482] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:58.989104 12182 raft_consensus.cc:504] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.989573 12182 leader_election.cc:288] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:56:58.990054 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91062abe57b451bb29aea0a20d5177b" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b" is_pre_election: true
I20260806 23:56:58.990062 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91062abe57b451bb29aea0a20d5177b" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" is_pre_election: true
I20260806 23:56:58.990229 12077 raft_consensus.cc:2454] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:56:58.990228 11874 raft_consensus.cc:2454] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:56:58.990579 11592 leader_election.cc:302] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91071d4612bb49b69c06b6d4f8feb8a3, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:56:58.990801 12182 raft_consensus.cc:2781] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:58.990871 12182 raft_consensus.cc:482] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:58.990900 12182 raft_consensus.cc:3037] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.991902 12182 raft_consensus.cc:504] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:58.992251 12182 leader_election.cc:288] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:56:58.992635 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91062abe57b451bb29aea0a20d5177b" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b"
I20260806 23:56:58.992772 11874 raft_consensus.cc:3037] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.992725 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91062abe57b451bb29aea0a20d5177b" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
I20260806 23:56:58.992854 12077 raft_consensus.cc:3037] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:58.993898 11874 raft_consensus.cc:2454] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:56:58.993906 12077 raft_consensus.cc:2454] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:56:58.994161 11592 leader_election.cc:302] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:56:58.994352 12182 raft_consensus.cc:2781] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:58.994455 12182 raft_consensus.cc:686] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [term 1 LEADER]: Becoming Leader. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Running, Role: LEADER
I20260806 23:56:58.994587 12182 consensus_queue.cc:237] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
W20260806 23:56:58.994992 12147 tablet.cc:2369] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:58.995959 12191 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: TxnStatusTablet state changed. Reason: New leader aa4c05eeb23b41888757b0ff22b603c6. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:58.996058 12191 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:58.996380 12210 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:56:58.996659 12210 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:56:58.996942 11446 catalog_manager.cc:5495] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:59.011245 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.007725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.8.204.65:41271)
0806 23:56:59.007848 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.007857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.007869 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:56:59.008030 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.008032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.008081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.008203 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.008460 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.008938 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.009765 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.009998 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.010109 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.010438 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.010885 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.011010 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.011079 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.011129 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:56:59.011260 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.007620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.0.0.1:57222)
0806 23:56:59.007774 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.007790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.007890 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.008218 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.008221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.008228 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:59.008348 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.008354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.009045 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.009050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.009620 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.009629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.010125 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.010129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.010241 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.010285 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.010986 (+ 701us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.011003 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.011070 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.011157 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 23:56:59.016191 12077 raft_consensus.cc:1261] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.016191 11874 raft_consensus.cc:1261] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.016848 12182 consensus_queue.cc:1035] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.017014 12191 consensus_queue.cc:1035] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.020247 12186 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: TxnStatusTablet state changed. Reason: New leader aa4c05eeb23b41888757b0ff22b603c6. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.020373 12186 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:59.020637 12187 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: TxnStatusTablet state changed. Reason: New leader aa4c05eeb23b41888757b0ff22b603c6. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.020720 12187 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:59.023856 12182 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.023973 12182 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:59.024076 12215 mvcc.cc:205] Tried to move back new op lower bound from 7315704295483858944 to 7315704295403466752. Current Snapshot: MvccSnapshot[applied={T|T < 7315704295483858944}]
I20260806 23:56:59.024649 12186 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.024761 12186 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 91071d4612bb49b69c06b6d4f8feb8a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:59.025538 12191 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.025640 12191 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P aa4c05eeb23b41888757b0ff22b603c6: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:56:59.026597 12217 mvcc.cc:205] Tried to move back new op lower bound from 7315704295483858944 to 7315704295403466752. Current Snapshot: MvccSnapshot[applied={T|T < 7315704295483858944}]
I20260806 23:56:59.027172 12187 tablet_replica.cc:439] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } } }
I20260806 23:56:59.027278 12187 tablet_replica.cc:442] T d91062abe57b451bb29aea0a20d5177b P 9a005f8104a343828ebe1edbb901f02b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:56:59.029063 12216 mvcc.cc:205] Tried to move back new op lower bound from 7315704295483858944 to 7315704295403466752. Current Snapshot: MvccSnapshot[applied={T|T < 7315704295483858944}]
W20260806 23:56:59.031320 11940 tablet.cc:2369] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:59.034621 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.031153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.8.204.65:41271)
0806 23:56:59.031256 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.031260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.031273 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:59.031315 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.031318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.031376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.031453 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.031730 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.032363 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.033602 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.033842 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.033920 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.034231 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.034291 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.034341 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.034494 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.034532 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:56:59.044162 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.041060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.8.204.65:40403)
0806 23:56:59.041396 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.041406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.041510 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.041719 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.041722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.041728 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:59.041815 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.041820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.042415 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.042419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.043067 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.043075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.043448 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.043451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.043546 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.043605 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.043922 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.043927 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.043978 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.044029 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":92,"threads_started":1}
I20260806 23:56:59.044181 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.041129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:40403 (local address 127.8.204.65:41271)
0806 23:56:59.041236 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.041240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.041470 (+ 230us) server_negotiation.cc:407] Connection header received
0806 23:56:59.041599 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.041603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.041641 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.041711 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.041910 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.042316 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.043173 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.043350 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.043447 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.043767 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.043833 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.043906 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.044016 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.044062 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:56:59.051296 12077 raft_consensus.cc:1261] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.051769 11874 raft_consensus.cc:1261] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.051782 12238 consensus_queue.cc:1035] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.052141 12238 consensus_queue.cc:1035] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.096195 11446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:59.096335 11446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:59.098399 11446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Sending DeleteTablet for 3 replicas of tablet 3acae4742e694d5bb61dd8c24c5c56b9
I20260806 23:56:59.099159 12044 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:59 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:41134
I20260806 23:56:59.099164 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:59 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60192
I20260806 23:56:59.099572 12243 tablet_replica.cc:330] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: stopping tablet replica
I20260806 23:56:59.099596 12244 tablet_replica.cc:330] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: stopping tablet replica
I20260806 23:56:59.099558 11653 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acae4742e694d5bb61dd8c24c5c56b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:59 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:57208
I20260806 23:56:59.099890 12244 raft_consensus.cc:2229] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:59.099905 12243 raft_consensus.cc:2229] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:59.100119 12243 raft_consensus.cc:2258] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:59.100119 12244 raft_consensus.cc:2258] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:59.100628 12245 tablet_replica.cc:330] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: stopping tablet replica
I20260806 23:56:59.100620 11446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:56:59.100935 12245 raft_consensus.cc:2229] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 LEADER]: Raft consensus shutting down.
W20260806 23:56:59.101032 11446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:59.101318 12245 raft_consensus.cc:2258] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:59.102150 12243 ts_tablet_manager.cc:1905] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:59.102581 12244 ts_tablet_manager.cc:1905] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:59.104410 12245 ts_tablet_manager.cc:1905] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:59.105475 12244 ts_tablet_manager.cc:1918] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:59.105623 12244 log.cc:1196] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/3acae4742e694d5bb61dd8c24c5c56b9
I20260806 23:56:59.105947 12244 ts_tablet_manager.cc:1939] T 3acae4742e694d5bb61dd8c24c5c56b9 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting consensus metadata
I20260806 23:56:59.106124 12044 tablet_service.cc:1467] Processing CreateTablet for tablet 05907d6f1b1d403ca98508a793f71d3e (DEFAULT_TABLE table=test-table [id=e744d7405ca44137997b11bda8724172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:59.106388 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05907d6f1b1d403ca98508a793f71d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:59.107524 11653 tablet_service.cc:1467] Processing CreateTablet for tablet 05907d6f1b1d403ca98508a793f71d3e (DEFAULT_TABLE table=test-table [id=e744d7405ca44137997b11bda8724172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:59.107787 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05907d6f1b1d403ca98508a793f71d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:59.108536 12177 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Bootstrap starting.
I20260806 23:56:59.109366 12243 ts_tablet_manager.cc:1918] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:59.109439 12177 tablet_bootstrap.cc:654] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:59.110097 12245 ts_tablet_manager.cc:1918] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:56:59.110129 11426 catalog_manager.cc:4841] TS 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409): tablet 3acae4742e694d5bb61dd8c24c5c56b9 (table test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]) successfully deleted
I20260806 23:56:59.110306 12245 log.cc:1196] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/3acae4742e694d5bb61dd8c24c5c56b9
I20260806 23:56:59.110442 12243 log.cc:1196] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/3acae4742e694d5bb61dd8c24c5c56b9
I20260806 23:56:59.110862 12176 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Bootstrap starting.
I20260806 23:56:59.110627 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 05907d6f1b1d403ca98508a793f71d3e (DEFAULT_TABLE table=test-table [id=e744d7405ca44137997b11bda8724172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:59.111284 12243 ts_tablet_manager.cc:1939] T 3acae4742e694d5bb61dd8c24c5c56b9 P 9a005f8104a343828ebe1edbb901f02b: Deleting consensus metadata
I20260806 23:56:59.110875 12245 ts_tablet_manager.cc:1939] T 3acae4742e694d5bb61dd8c24c5c56b9 P aa4c05eeb23b41888757b0ff22b603c6: Deleting consensus metadata
I20260806 23:56:59.111361 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05907d6f1b1d403ca98508a793f71d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:59.111902 12176 tablet_bootstrap.cc:654] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:59.113083 11426 catalog_manager.cc:4841] TS 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155): tablet 3acae4742e694d5bb61dd8c24c5c56b9 (table test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]) successfully deleted
I20260806 23:56:59.113476 11426 catalog_manager.cc:4841] TS aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271): tablet 3acae4742e694d5bb61dd8c24c5c56b9 (table test-table [id=2aa475d9a3a04b67b9a941cf70057ae5]) successfully deleted
I20260806 23:56:59.113803 12175 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Bootstrap starting.
I20260806 23:56:59.114418 12176 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: No bootstrap required, opened a new log
I20260806 23:56:59.114429 12177 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: No bootstrap required, opened a new log
I20260806 23:56:59.114498 12176 ts_tablet_manager.cc:1397] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 23:56:59.114498 12177 ts_tablet_manager.cc:1397] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260806 23:56:59.114593 12175 tablet_bootstrap.cc:654] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:59.114928 12176 raft_consensus.cc:348] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.114928 12177 raft_consensus.cc:348] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.115026 12177 raft_consensus.cc:374] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:59.115026 12176 raft_consensus.cc:374] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:59.115058 12176 raft_consensus.cc:729] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Initialized, Role: FOLLOWER
I20260806 23:56:59.115058 12177 raft_consensus.cc:729] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91071d4612bb49b69c06b6d4f8feb8a3, State: Initialized, Role: FOLLOWER
I20260806 23:56:59.115173 12176 consensus_queue.cc:260] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.115173 12177 consensus_queue.cc:260] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.115406 12177 ts_tablet_manager.cc:1428] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:56:59.115406 12176 ts_tablet_manager.cc:1428] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:56:59.116228 12175 tablet_bootstrap.cc:492] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: No bootstrap required, opened a new log
I20260806 23:56:59.116312 12175 ts_tablet_manager.cc:1397] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 23:56:59.116750 12175 raft_consensus.cc:348] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.116865 12175 raft_consensus.cc:374] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:59.116899 12175 raft_consensus.cc:729] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Initialized, Role: FOLLOWER
I20260806 23:56:59.117012 12175 consensus_queue.cc:260] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.117238 12175 ts_tablet_manager.cc:1428] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:56:59.242223 12187 raft_consensus.cc:482] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:59.242383 12187 raft_consensus.cc:504] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.243659 12187 leader_election.cc:288] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:56:59.248167 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.243999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:37079 (local address 127.8.204.65:41271)
0806 23:56:59.244134 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.244139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.244275 (+ 136us) server_negotiation.cc:407] Connection header received
0806 23:56:59.244552 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.244555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.244615 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.244670 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:59.245160 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.245792 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.246995 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.247392 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.247529 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.247609 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.247948 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.248023 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:56:59.248200 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.243887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.8.204.66:37079)
0806 23:56:59.244176 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.244193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.244378 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.244852 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.244858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.244878 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:59.245003 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.245008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.245940 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.245943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.246871 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.246882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.247681 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.247684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.247756 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.247775 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.247927 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.247999 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260806 23:56:59.248944 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.244307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:37409 (local address 127.8.204.66:34657)
0806 23:56:59.244717 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.244730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.244826 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.245097 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.245103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.245128 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:59.245248 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.245254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.246240 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.246243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.247455 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.247467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.248473 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.248477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.248577 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.248611 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.248722 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.248791 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260806 23:56:59.249035 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.244450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:34657 (local address 127.8.204.67:37409)
0806 23:56:59.244573 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.244578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.244813 (+ 235us) server_negotiation.cc:407] Connection header received
0806 23:56:59.244916 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.244919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.244975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.245064 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:59.245368 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.246101 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.247606 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.248200 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.248295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.248341 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.248771 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.248868 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:56:59.249050 11683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05907d6f1b1d403ca98508a793f71d3e" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4c05eeb23b41888757b0ff22b603c6" is_pre_election: true
I20260806 23:56:59.249303 11683 raft_consensus.cc:2454] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 0.
I20260806 23:56:59.249418 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05907d6f1b1d403ca98508a793f71d3e" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" is_pre_election: true
I20260806 23:56:59.249567 12077 raft_consensus.cc:2454] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 0.
I20260806 23:56:59.249754 11781 leader_election.cc:302] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:56:59.250008 12187 raft_consensus.cc:2781] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:59.250085 12187 raft_consensus.cc:482] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:59.250135 12187 raft_consensus.cc:3037] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:59.251235 12187 raft_consensus.cc:504] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.251567 12187 leader_election.cc:288] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 election: Requested vote from peers aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:56:59.251986 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05907d6f1b1d403ca98508a793f71d3e" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
I20260806 23:56:59.252014 11683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05907d6f1b1d403ca98508a793f71d3e" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4c05eeb23b41888757b0ff22b603c6"
I20260806 23:56:59.252110 11683 raft_consensus.cc:3037] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:59.252113 12077 raft_consensus.cc:3037] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:59.253188 11683 raft_consensus.cc:2454] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 1.
I20260806 23:56:59.253188 12077 raft_consensus.cc:2454] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 1.
I20260806 23:56:59.253495 11781 leader_election.cc:302] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91071d4612bb49b69c06b6d4f8feb8a3, 9a005f8104a343828ebe1edbb901f02b; no voters:
I20260806 23:56:59.253682 12187 raft_consensus.cc:2781] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:59.253931 12187 raft_consensus.cc:686] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 1 LEADER]: Becoming Leader. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Running, Role: LEADER
I20260806 23:56:59.254177 12187 consensus_queue.cc:237] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:56:59.256672 11448 catalog_manager.cc:5495] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a005f8104a343828ebe1edbb901f02b (127.8.204.66). New cstate: current_term: 1 leader_uuid: "9a005f8104a343828ebe1edbb901f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:59.279702 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.275736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.8.204.66:44155)
0806 23:56:59.275844 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.275848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.275862 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:56:59.276015 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.276018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.276078 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.276194 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.276399 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.277172 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.277948 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.278348 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.278463 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.278757 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.279320 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.279444 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.279504 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.279568 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:56:59.289786 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.286538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:40881 (local address 127.8.204.65:41271)
0806 23:56:59.286668 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.286672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.286864 (+ 192us) server_negotiation.cc:407] Connection header received
0806 23:56:59.286943 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.286946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.286993 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.287076 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.287281 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.287949 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.288705 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.289042 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.289143 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.289378 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.289451 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.289517 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.289594 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.289640 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:56:59.290019 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.286386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.8.204.66:40881)
0806 23:56:59.286778 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.286792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.286881 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.287097 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.287099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.287104 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:59.287190 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.287194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.288051 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.288053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.288589 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.288596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.289151 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.289154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.289220 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.289259 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.289529 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.289534 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.289577 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.289634 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260806 23:56:59.296214 12077 raft_consensus.cc:1261] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Refusing update from remote peer 9a005f8104a343828ebe1edbb901f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.296314 11683 raft_consensus.cc:1261] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Refusing update from remote peer 9a005f8104a343828ebe1edbb901f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:59.296712 12252 consensus_queue.cc:1035] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.296872 12187 consensus_queue.cc:1035] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:59.314903 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.311087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44155 (local address 127.8.204.65:44859)
0806 23:56:59.311241 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:59.311254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:59.311320 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:59.312294 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:59.312296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:59.312302 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:56:59.312383 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.312387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.313098 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.313100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.313815 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:59.313824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.314240 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.314242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:59.314301 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.314338 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.314616 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.314622 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:59.314684 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:56:59.314782 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 23:56:59.314934 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:59.311226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:44859 (local address 127.8.204.66:44155)
0806 23:56:59.312104 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:59.312110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:59.312124 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:56:59.312164 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:59.312167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:59.312210 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:59.312284 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:59.312494 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.313005 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.313916 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:59.314109 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:59.314232 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:59.314448 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:59.314523 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:59.314589 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:59.314759 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 23:56:59.314810 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777}
W20260806 23:56:59.315610 12216 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 23:57:00.873289 11448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:00.873428 11448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:00.875578 11448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Sending DeleteTablet for 3 replicas of tablet 05907d6f1b1d403ca98508a793f71d3e
I20260806 23:57:00.876088 11653 tablet_service.cc:1514] Processing DeleteTablet for tablet 05907d6f1b1d403ca98508a793f71d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:57208
I20260806 23:57:00.876168 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 05907d6f1b1d403ca98508a793f71d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60192
I20260806 23:57:00.876223 12044 tablet_service.cc:1514] Processing DeleteTablet for tablet 05907d6f1b1d403ca98508a793f71d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:41134
I20260806 23:57:00.876560 12299 tablet_replica.cc:330] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: stopping tablet replica
I20260806 23:57:00.876554 12298 tablet_replica.cc:330] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: stopping tablet replica
I20260806 23:57:00.876704 12300 tablet_replica.cc:330] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: stopping tablet replica
I20260806 23:57:00.876731 12299 raft_consensus.cc:2229] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:00.876739 12298 raft_consensus.cc:2229] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:00.876853 12298 raft_consensus.cc:2258] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:00.877027 12299 raft_consensus.cc:2258] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:00.876883 12300 raft_consensus.cc:2229] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:00.877327 12300 raft_consensus.cc:2258] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:00.877512 12298 ts_tablet_manager.cc:1905] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:00.877643 12299 ts_tablet_manager.cc:1905] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:00.879269 12300 ts_tablet_manager.cc:1905] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:00.880463 12299 ts_tablet_manager.cc:1918] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:57:00.880549 12299 log.cc:1196] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/05907d6f1b1d403ca98508a793f71d3e
I20260806 23:57:00.880659 12298 ts_tablet_manager.cc:1918] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:57:00.880762 12298 log.cc:1196] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/05907d6f1b1d403ca98508a793f71d3e
I20260806 23:57:00.880890 12299 ts_tablet_manager.cc:1939] T 05907d6f1b1d403ca98508a793f71d3e P 9a005f8104a343828ebe1edbb901f02b: Deleting consensus metadata
I20260806 23:57:00.881074 12298 ts_tablet_manager.cc:1939] T 05907d6f1b1d403ca98508a793f71d3e P aa4c05eeb23b41888757b0ff22b603c6: Deleting consensus metadata
I20260806 23:57:00.881644 11426 catalog_manager.cc:4841] TS 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155): tablet 05907d6f1b1d403ca98508a793f71d3e (table test-table [id=e744d7405ca44137997b11bda8724172]) successfully deleted
I20260806 23:57:00.881844 11426 catalog_manager.cc:4841] TS aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271): tablet 05907d6f1b1d403ca98508a793f71d3e (table test-table [id=e744d7405ca44137997b11bda8724172]) successfully deleted
I20260806 23:57:00.882021 12300 ts_tablet_manager.cc:1918] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:57:00.882105 12300 log.cc:1196] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/05907d6f1b1d403ca98508a793f71d3e
I20260806 23:57:00.882372 12300 ts_tablet_manager.cc:1939] T 05907d6f1b1d403ca98508a793f71d3e P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting consensus metadata
I20260806 23:57:00.883013 11426 catalog_manager.cc:4841] TS 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409): tablet 05907d6f1b1d403ca98508a793f71d3e (table test-table [id=e744d7405ca44137997b11bda8724172]) successfully deleted
Aug 06 23:57:00 dist-test-slave-39l8 krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:57:00 dist-test-slave-39l8 krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:00.905143 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:00.897380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.8.204.126:42383)
0806 23:57:00.897645 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:00.897651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:00.897666 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:00.897798 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:00.897800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:00.897840 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:00.897911 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:00.898480 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:00.899057 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:00.899757 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:00.899981 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:00.900056 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:00.902238 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:00.902241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:00.902242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:00.902263 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:00.903646 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:00.904055 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:00.904057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:00.904058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:00.904176 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:00.904396 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:00.904398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:00.904399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:00.904549 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:00.904641 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:00.904854 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 23:57:00.904959 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260806 23:57:00.907526 11448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:00.907950 11448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:00.912274 11653 tablet_service.cc:1467] Processing CreateTablet for tablet 31930054540643e799ff2f62dedd13de (DEFAULT_TABLE table=test-table [id=8294b118ac8449c084af3059250f7598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:00.912300 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 31930054540643e799ff2f62dedd13de (DEFAULT_TABLE table=test-table [id=8294b118ac8449c084af3059250f7598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:00.912560 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31930054540643e799ff2f62dedd13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:00.912499 12044 tablet_service.cc:1467] Processing CreateTablet for tablet 31930054540643e799ff2f62dedd13de (DEFAULT_TABLE table=test-table [id=8294b118ac8449c084af3059250f7598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:00.912560 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31930054540643e799ff2f62dedd13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:00.912793 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31930054540643e799ff2f62dedd13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:00.915205 12316 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Bootstrap starting.
I20260806 23:57:00.915264 12318 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Bootstrap starting.
I20260806 23:57:00.915413 12317 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Bootstrap starting.
I20260806 23:57:00.916003 12318 tablet_bootstrap.cc:654] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:00.916034 12316 tablet_bootstrap.cc:654] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:00.916244 12317 tablet_bootstrap.cc:654] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:00.917179 12316 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: No bootstrap required, opened a new log
I20260806 23:57:00.917181 12318 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: No bootstrap required, opened a new log
I20260806 23:57:00.917248 12316 ts_tablet_manager.cc:1397] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:00.917268 12318 ts_tablet_manager.cc:1397] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:00.917626 12316 raft_consensus.cc:348] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.917734 12316 raft_consensus.cc:374] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:00.917766 12316 raft_consensus.cc:729] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Initialized, Role: FOLLOWER
I20260806 23:57:00.917708 12318 raft_consensus.cc:348] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.917819 12318 raft_consensus.cc:374] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:00.917852 12318 raft_consensus.cc:729] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91071d4612bb49b69c06b6d4f8feb8a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:00.917882 12316 consensus_queue.cc:260] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.917973 12318 consensus_queue.cc:260] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.918177 12316 ts_tablet_manager.cc:1428] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:00.918347 12318 ts_tablet_manager.cc:1428] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:00.918848 12317 tablet_bootstrap.cc:492] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: No bootstrap required, opened a new log
I20260806 23:57:00.918926 12317 ts_tablet_manager.cc:1397] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 23:57:00.919342 12317 raft_consensus.cc:348] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.919448 12317 raft_consensus.cc:374] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:00.919489 12317 raft_consensus.cc:729] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Initialized, Role: FOLLOWER
I20260806 23:57:00.919634 12317 consensus_queue.cc:260] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.919935 12317 ts_tablet_manager.cc:1428] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:57:00.971921 12238 raft_consensus.cc:482] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:00.972084 12238 raft_consensus.cc:504] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.972460 12238 leader_election.cc:288] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:57:00.972906 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31930054540643e799ff2f62dedd13de" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b" is_pre_election: true
I20260806 23:57:00.972934 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31930054540643e799ff2f62dedd13de" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" is_pre_election: true
I20260806 23:57:00.973062 12077 raft_consensus.cc:2454] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:57:00.973062 11874 raft_consensus.cc:2454] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 0.
I20260806 23:57:00.973393 11592 leader_election.cc:302] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:57:00.973577 12238 raft_consensus.cc:2781] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:00.973639 12238 raft_consensus.cc:482] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:00.973686 12238 raft_consensus.cc:3037] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:00.974674 12238 raft_consensus.cc:504] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.975030 12238 leader_election.cc:288] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:57:00.975392 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31930054540643e799ff2f62dedd13de" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a005f8104a343828ebe1edbb901f02b"
I20260806 23:57:00.975461 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31930054540643e799ff2f62dedd13de" candidate_uuid: "aa4c05eeb23b41888757b0ff22b603c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
I20260806 23:57:00.975515 11874 raft_consensus.cc:3037] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:00.975603 12077 raft_consensus.cc:3037] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:00.976586 11874 raft_consensus.cc:2454] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:57:00.976635 12077 raft_consensus.cc:2454] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4c05eeb23b41888757b0ff22b603c6 in term 1.
I20260806 23:57:00.976882 11592 leader_election.cc:302] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:57:00.977041 12238 raft_consensus.cc:2781] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:00.977149 12238 raft_consensus.cc:686] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 1 LEADER]: Becoming Leader. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Running, Role: LEADER
I20260806 23:57:00.977284 12238 consensus_queue.cc:237] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:00.978860 11446 catalog_manager.cc:5495] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "aa4c05eeb23b41888757b0ff22b603c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:01.002996 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:00.999820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.8.204.65:41271)
0806 23:57:01.000120 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:01.000129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:01.000149 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:57:01.000185 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:01.000187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:01.000226 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:01.000319 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:01.000577 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:01.001111 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:01.001949 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:01.002168 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:01.002270 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:01.002537 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:01.002601 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:01.002678 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:01.002809 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 23:57:01.002859 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":123,"threads_started":1}
I20260806 23:57:01.008514 12077 raft_consensus.cc:1261] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:01.008515 11874 raft_consensus.cc:1261] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Refusing update from remote peer aa4c05eeb23b41888757b0ff22b603c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:01.008917 12271 consensus_queue.cc:1035] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:01.009109 12277 consensus_queue.cc:1035] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:01.038971 11446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:01.039096 11446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:01.040797 11446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Sending DeleteTablet for 3 replicas of tablet 31930054540643e799ff2f62dedd13de
I20260806 23:57:01.041388 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 31930054540643e799ff2f62dedd13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:01 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60192
I20260806 23:57:01.041471 11653 tablet_service.cc:1514] Processing DeleteTablet for tablet 31930054540643e799ff2f62dedd13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:01 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:57208
I20260806 23:57:01.041461 12044 tablet_service.cc:1514] Processing DeleteTablet for tablet 31930054540643e799ff2f62dedd13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:01 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:41134
I20260806 23:57:01.041561 12299 tablet_replica.cc:330] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: stopping tablet replica
I20260806 23:57:01.041672 12298 tablet_replica.cc:330] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: stopping tablet replica
I20260806 23:57:01.041668 12299 raft_consensus.cc:2229] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:01.041800 12299 raft_consensus.cc:2258] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:01.041808 12298 raft_consensus.cc:2229] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:01.042004 12298 raft_consensus.cc:2258] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:01.042008 12300 tablet_replica.cc:330] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: stopping tablet replica
I20260806 23:57:01.042119 12300 raft_consensus.cc:2229] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:01.042232 12300 raft_consensus.cc:2258] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:01.042603 12298 ts_tablet_manager.cc:1905] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:01.043002 12299 ts_tablet_manager.cc:1905] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:01.042995 11446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:01.043397 11446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:01.045344 12298 ts_tablet_manager.cc:1918] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:01.045445 12298 log.cc:1196] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/31930054540643e799ff2f62dedd13de
I20260806 23:57:01.045421 12299 ts_tablet_manager.cc:1918] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:01.045506 12299 log.cc:1196] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/31930054540643e799ff2f62dedd13de
I20260806 23:57:01.045745 12298 ts_tablet_manager.cc:1939] T 31930054540643e799ff2f62dedd13de P aa4c05eeb23b41888757b0ff22b603c6: Deleting consensus metadata
I20260806 23:57:01.045835 12299 ts_tablet_manager.cc:1939] T 31930054540643e799ff2f62dedd13de P 9a005f8104a343828ebe1edbb901f02b: Deleting consensus metadata
I20260806 23:57:01.046468 11426 catalog_manager.cc:4841] TS aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271): tablet 31930054540643e799ff2f62dedd13de (table test-table [id=8294b118ac8449c084af3059250f7598]) successfully deleted
I20260806 23:57:01.046622 11426 catalog_manager.cc:4841] TS 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155): tablet 31930054540643e799ff2f62dedd13de (table test-table [id=8294b118ac8449c084af3059250f7598]) successfully deleted
I20260806 23:57:01.046877 12300 ts_tablet_manager.cc:1905] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:01.048254 11653 tablet_service.cc:1467] Processing CreateTablet for tablet 82d7b27a7427412d87f675105b12ee77 (DEFAULT_TABLE table=test-table [id=8f283286723f4b0a84b838da340b73f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:01.048303 12044 tablet_service.cc:1467] Processing CreateTablet for tablet 82d7b27a7427412d87f675105b12ee77 (DEFAULT_TABLE table=test-table [id=8f283286723f4b0a84b838da340b73f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:01.048333 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 82d7b27a7427412d87f675105b12ee77 (DEFAULT_TABLE table=test-table [id=8f283286723f4b0a84b838da340b73f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:01.048508 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d7b27a7427412d87f675105b12ee77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:01.048527 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d7b27a7427412d87f675105b12ee77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:01.048564 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d7b27a7427412d87f675105b12ee77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:01.049404 12300 ts_tablet_manager.cc:1918] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:01.049467 12300 log.cc:1196] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/31930054540643e799ff2f62dedd13de
I20260806 23:57:01.049723 12300 ts_tablet_manager.cc:1939] T 31930054540643e799ff2f62dedd13de P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting consensus metadata
I20260806 23:57:01.050642 12318 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Bootstrap starting.
I20260806 23:57:01.050855 12317 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: Bootstrap starting.
I20260806 23:57:01.051146 11426 catalog_manager.cc:4841] TS 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409): tablet 31930054540643e799ff2f62dedd13de (table test-table [id=8294b118ac8449c084af3059250f7598]) successfully deleted
I20260806 23:57:01.051607 12317 tablet_bootstrap.cc:654] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:01.051641 12318 tablet_bootstrap.cc:654] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:01.052191 12316 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Bootstrap starting.
I20260806 23:57:01.052816 12316 tablet_bootstrap.cc:654] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:01.053885 12316 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: No bootstrap required, opened a new log
I20260806 23:57:01.053960 12316 ts_tablet_manager.cc:1397] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:01.054122 12318 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: No bootstrap required, opened a new log
I20260806 23:57:01.054209 12318 ts_tablet_manager.cc:1397] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 23:57:01.054311 12316 raft_consensus.cc:348] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.054399 12316 raft_consensus.cc:374] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:01.054425 12316 raft_consensus.cc:729] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Initialized, Role: FOLLOWER
I20260806 23:57:01.054526 12316 consensus_queue.cc:260] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.054638 12318 raft_consensus.cc:348] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.054778 12318 raft_consensus.cc:374] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:01.054790 12316 ts_tablet_manager.cc:1428] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:01.054816 12318 raft_consensus.cc:729] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91071d4612bb49b69c06b6d4f8feb8a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:01.054867 12317 tablet_bootstrap.cc:492] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: No bootstrap required, opened a new log
I20260806 23:57:01.054944 12317 ts_tablet_manager.cc:1397] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 23:57:01.054929 12318 consensus_queue.cc:260] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.055169 12318 ts_tablet_manager.cc:1428] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:01.055317 12317 raft_consensus.cc:348] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.055454 12317 raft_consensus.cc:374] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:01.055495 12317 raft_consensus.cc:729] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4c05eeb23b41888757b0ff22b603c6, State: Initialized, Role: FOLLOWER
I20260806 23:57:01.055606 12317 consensus_queue.cc:260] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.055863 12317 ts_tablet_manager.cc:1428] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:01.270679 12292 raft_consensus.cc:482] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:01.270857 12292 raft_consensus.cc:504] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.271327 12292 leader_election.cc:288] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:57:01.271788 11683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d7b27a7427412d87f675105b12ee77" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4c05eeb23b41888757b0ff22b603c6" is_pre_election: true
I20260806 23:57:01.271860 12076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d7b27a7427412d87f675105b12ee77" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" is_pre_election: true
I20260806 23:57:01.271946 11683 raft_consensus.cc:2454] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 0.
I20260806 23:57:01.272082 12076 raft_consensus.cc:2454] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 0.
I20260806 23:57:01.272368 11781 leader_election.cc:302] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:57:01.272562 12292 raft_consensus.cc:2781] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:01.272629 12292 raft_consensus.cc:482] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:01.272657 12292 raft_consensus.cc:3037] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:01.273669 12292 raft_consensus.cc:504] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.274035 12292 leader_election.cc:288] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 election: Requested vote from peers aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271), 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409)
I20260806 23:57:01.274394 11683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d7b27a7427412d87f675105b12ee77" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4c05eeb23b41888757b0ff22b603c6"
I20260806 23:57:01.274519 11683 raft_consensus.cc:3037] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:01.274495 12076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d7b27a7427412d87f675105b12ee77" candidate_uuid: "9a005f8104a343828ebe1edbb901f02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91071d4612bb49b69c06b6d4f8feb8a3"
I20260806 23:57:01.274631 12076 raft_consensus.cc:3037] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:01.275475 11683 raft_consensus.cc:2454] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 1.
I20260806 23:57:01.275553 12076 raft_consensus.cc:2454] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a005f8104a343828ebe1edbb901f02b in term 1.
I20260806 23:57:01.275739 11781 leader_election.cc:302] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a005f8104a343828ebe1edbb901f02b, aa4c05eeb23b41888757b0ff22b603c6; no voters:
I20260806 23:57:01.275907 12292 raft_consensus.cc:2781] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:01.276007 12292 raft_consensus.cc:686] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 1 LEADER]: Becoming Leader. State: Replica: 9a005f8104a343828ebe1edbb901f02b, State: Running, Role: LEADER
I20260806 23:57:01.276165 12292 consensus_queue.cc:237] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } }
I20260806 23:57:01.277684 11450 catalog_manager.cc:5495] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a005f8104a343828ebe1edbb901f02b (127.8.204.66). New cstate: current_term: 1 leader_uuid: "9a005f8104a343828ebe1edbb901f02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a005f8104a343828ebe1edbb901f02b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:01.295392 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:01.291990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.8.204.66:44155)
0806 23:57:01.292271 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:01.292290 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:01.292304 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:01.292354 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:01.292356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:01.292393 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:01.292482 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:01.292733 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:01.293341 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:01.294324 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:01.294547 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:01.294633 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:01.294935 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:01.295002 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:01.295061 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:01.295172 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 23:57:01.295217 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260806 23:57:01.301036 12076 raft_consensus.cc:1261] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Refusing update from remote peer 9a005f8104a343828ebe1edbb901f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:01.301040 11683 raft_consensus.cc:1261] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Refusing update from remote peer 9a005f8104a343828ebe1edbb901f02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:01.301420 12292 consensus_queue.cc:1035] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91071d4612bb49b69c06b6d4f8feb8a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:01.301563 12284 consensus_queue.cc:1035] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa4c05eeb23b41888757b0ff22b603c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:57:01.313319 12272 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 23:57:02.859771 11450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:02.859967 11450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:02.862224 11450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71bce6c686344781b19b2a95f322b7d1: Sending DeleteTablet for 3 replicas of tablet 82d7b27a7427412d87f675105b12ee77
I20260806 23:57:02.862746 11653 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d7b27a7427412d87f675105b12ee77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:02 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:57208
I20260806 23:57:02.862794 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d7b27a7427412d87f675105b12ee77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:02 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60192
I20260806 23:57:02.862847 12044 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d7b27a7427412d87f675105b12ee77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:02 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:41134
I20260806 23:57:02.863099 12370 tablet_replica.cc:330] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6: stopping tablet replica
I20260806 23:57:02.863250 12372 tablet_replica.cc:330] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: stopping tablet replica
I20260806 23:57:02.863271 12370 raft_consensus.cc:2229] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:02.863391 12370 raft_consensus.cc:2258] T 82d7b27a7427412d87f675105b12ee77 P aa4c05eeb23b41888757b0ff22b603c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:02.863451 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 11562
I20260806 23:57:02.863444 12372 raft_consensus.cc:2229] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:02.863359 12374 tablet_replica.cc:330] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: stopping tablet replica
I20260806 23:57:02.864286 12374 raft_consensus.cc:2229] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:02.864388 12374 raft_consensus.cc:2258] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:02.864454 12372 raft_consensus.cc:2258] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:02.864948 12374 ts_tablet_manager.cc:1905] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:02.866410 12372 ts_tablet_manager.cc:1905] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:02.867820 12374 ts_tablet_manager.cc:1918] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:57:02.867913 12374 log.cc:1196] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/82d7b27a7427412d87f675105b12ee77
I20260806 23:57:02.868255 12374 ts_tablet_manager.cc:1939] T 82d7b27a7427412d87f675105b12ee77 P 91071d4612bb49b69c06b6d4f8feb8a3: Deleting consensus metadata
I20260806 23:57:02.868870 11426 catalog_manager.cc:4841] TS 91071d4612bb49b69c06b6d4f8feb8a3 (127.8.204.67:37409): tablet 82d7b27a7427412d87f675105b12ee77 (table test-table [id=8f283286723f4b0a84b838da340b73f7]) successfully deleted
I20260806 23:57:02.869858 12372 ts_tablet_manager.cc:1918] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:57:02.869947 12372 log.cc:1196] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/82d7b27a7427412d87f675105b12ee77
I20260806 23:57:02.870419 12372 ts_tablet_manager.cc:1939] T 82d7b27a7427412d87f675105b12ee77 P 9a005f8104a343828ebe1edbb901f02b: Deleting consensus metadata
I20260806 23:57:02.871949 11426 catalog_manager.cc:4841] TS 9a005f8104a343828ebe1edbb901f02b (127.8.204.66:44155): tablet 82d7b27a7427412d87f675105b12ee77 (table test-table [id=8f283286723f4b0a84b838da340b73f7]) successfully deleted
W20260806 23:57:02.880089 11426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:41271 (error 108)
I20260806 23:57:02.880326 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 11756
I20260806 23:57:02.889354 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:02.883784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41271 (local address 127.0.0.1:58180)
0806 23:57:02.889203 (+ 5419us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:02.889271 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5334,"thread_start_us":467,"threads_started":1}
W20260806 23:57:02.889539 11426 catalog_manager.cc:4584] TS aa4c05eeb23b41888757b0ff22b603c6 (127.8.204.65:41271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82d7b27a7427412d87f675105b12ee77: Network error: Client connection negotiation failed: client connection to 127.8.204.65:41271: connect: Connection refused (error 111)
I20260806 23:57:02.895781 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 11953
I20260806 23:57:02.904724 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 11396
2026-08-06T23:57:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6314 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:02 dist-test-slave-39l8 krb5kdc[12383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:02 dist-test-slave-39l8 krb5kdc[12383](info): set up 2 sockets
Aug 06 23:57:02 dist-test-slave-39l8 krb5kdc[12383](info): commencing operation
krb5kdc: starting...
W20260806 23:57:04.061278 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:04.175714 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45899
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:45899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:04.277529 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:04.277776 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:04.277865 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:04.281436 12400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:04.281512 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:04.281563 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:04.281631 12400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:04.281665 12400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:04.285934 12400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:45899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:04.286800 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:04.287613 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:04.295471 12408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:04.295516 12406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.295620 12400 server_base.cc:1034] running on GCE node
W20260806 23:57:04.295516 12411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.296254 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:04.297202 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:04.298389 12400 hybrid_clock.cc:648] HybridClock initialized: now 1786060624298367 us; error 33 us; skew 500 ppm
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:04.304481 12400 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:04.305279 12400 webserver.cc:460] Webserver started at http://127.8.204.126:41883/ using document root <none> and password file <none>
I20260806 23:57:04.305811 12400 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:04.305899 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:04.306125 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:04.308139 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "ba4301218ea849c6a12ee9f1b3000a67"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "fe3dbeff1c7df62968fea987d28d0f57"
server_key_iv: "1e613faf9f290ed10ddd6944aabfeb0d"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.308715 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "ba4301218ea849c6a12ee9f1b3000a67"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "fe3dbeff1c7df62968fea987d28d0f57"
server_key_iv: "1e613faf9f290ed10ddd6944aabfeb0d"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.312302 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:04.314510 12420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.315623 12400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:04.315753 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "ba4301218ea849c6a12ee9f1b3000a67"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "fe3dbeff1c7df62968fea987d28d0f57"
server_key_iv: "1e613faf9f290ed10ddd6944aabfeb0d"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.315850 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:04.355573 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:04.358258 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:04.358498 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:04.365774 12400 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:45899
I20260806 23:57:04.365788 12497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:45899 every 8 connection(s)
I20260806 23:57:04.366778 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:04.369457 12498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:04.371673 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 12400
I20260806 23:57:04.371768 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:04.372066 9009 external_mini_cluster.cc:1428] Setting key d41794d53657dc0342d483adf8a7257d
I20260806 23:57:04.375730 12498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67: Bootstrap starting.
I20260806 23:57:04.378280 12498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:04.379206 12498 log.cc:824] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:04.381091 12498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67: No bootstrap required, opened a new log
I20260806 23:57:04.384552 12498 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } }
I20260806 23:57:04.384788 12498 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:04.384886 12498 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4301218ea849c6a12ee9f1b3000a67, State: Initialized, Role: FOLLOWER
I20260806 23:57:04.385426 12498 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } }
I20260806 23:57:04.385586 12498 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:04.385685 12498 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:04.385797 12498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:04.385968 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:04.373470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.8.204.126:45899)
0806 23:57:04.373869 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:04.373887 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:04.373927 (+ 40us) server_negotiation.cc:407] Connection header received
0806 23:57:04.374713 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:04.374735 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:04.375139 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:04.375459 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:04.376144 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.377487 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.378266 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.378969 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.379164 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:04.381519 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:04.381524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:04.381535 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:04.381561 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:04.383128 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.383541 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.383547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.383552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.383656 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.383939 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.383942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.383944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.384295 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:04.384537 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:04.384836 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 23:57:04.385125 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260806 23:57:04.386652 12498 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } }
I20260806 23:57:04.387012 12498 leader_election.cc:302] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4301218ea849c6a12ee9f1b3000a67; no voters:
I20260806 23:57:04.387318 12498 leader_election.cc:288] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:04.387426 12504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:04.387642 12504 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [term 1 LEADER]: Becoming Leader. State: Replica: ba4301218ea849c6a12ee9f1b3000a67, State: Running, Role: LEADER
I20260806 23:57:04.387943 12504 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } }
I20260806 23:57:04.388417 12498 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:04.389014 12504 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4301218ea849c6a12ee9f1b3000a67. Latest consensus state: current_term: 1 leader_uuid: "ba4301218ea849c6a12ee9f1b3000a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } } }
I20260806 23:57:04.389211 12504 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:04.389505 12505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4301218ea849c6a12ee9f1b3000a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4301218ea849c6a12ee9f1b3000a67" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45899 } } }
I20260806 23:57:04.389662 12521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:04.389693 12505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:04.392748 12521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:04.397680 12521 catalog_manager.cc:1347] Generated new cluster ID: b8b77b8c0ce04e0b9121d13317b7fee2
I20260806 23:57:04.397737 12521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:04.416738 12521 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:04.417800 12521 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:04.422408 12521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4301218ea849c6a12ee9f1b3000a67: Generated new TSK 0
I20260806 23:57:04.423046 12521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:04.456677 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:04.560364 12536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:04.560575 12536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:04.560616 12536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:04.564886 12536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:04.564932 12536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:04.564998 12536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:04.569506 12536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:04.570346 12536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:04.571327 12536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:04.579911 12542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:04.579917 12544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.579948 12536 server_base.cc:1034] running on GCE node
W20260806 23:57:04.579937 12547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.580556 12536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:04.581182 12536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:04.582337 12536 hybrid_clock.cc:648] HybridClock initialized: now 1786060624582306 us; error 43 us; skew 500 ppm
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:04.584959 12536 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:04.585744 12536 webserver.cc:460] Webserver started at http://127.8.204.65:35827/ using document root <none> and password file <none>
I20260806 23:57:04.586277 12536 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:04.586364 12536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:04.586617 12536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:04.588656 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "9fdc19e9f2504f9e921dc8a74032e46d"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "5e4f8c9180fdea3ce7a089b8a58b1a53"
server_key_iv: "f1ee93cca4f0da922c698fc4f349e630"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.589231 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "9fdc19e9f2504f9e921dc8a74032e46d"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "5e4f8c9180fdea3ce7a089b8a58b1a53"
server_key_iv: "f1ee93cca4f0da922c698fc4f349e630"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.592760 12536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:04.594871 12557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.595899 12536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:04.596027 12536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "9fdc19e9f2504f9e921dc8a74032e46d"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "5e4f8c9180fdea3ce7a089b8a58b1a53"
server_key_iv: "f1ee93cca4f0da922c698fc4f349e630"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.596124 12536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:04.610262 12536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:04.613332 12536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:04.613504 12536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:04.613999 12536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:04.614748 12536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:04.614820 12536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.614877 12536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:04.614921 12536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.623732 12536 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:41103
I20260806 23:57:04.623826 12722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:41103 every 8 connection(s)
I20260806 23:57:04.624568 12536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:04.632310 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 12536
I20260806 23:57:04.632441 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:04.632704 9009 external_mini_cluster.cc:1428] Setting key 7465a6bbaad7c016cd8aa3928fa13079
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:04.637022 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:04.626291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:34303 (local address 127.8.204.126:45899)
0806 23:57:04.626442 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:04.626447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:04.627462 (+ 1015us) server_negotiation.cc:407] Connection header received
0806 23:57:04.628145 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:04.628148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:04.628191 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:04.628259 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:04.629197 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.629694 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.630700 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.630985 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.631067 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:04.633376 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:04.633379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:04.633381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:04.633404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:04.635047 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.635534 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.635537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.635538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.635622 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.635997 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.636003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.636006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.636198 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:04.636275 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:04.636828 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 23:57:04.636917 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:57:04.637889 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:04.626676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45899 (local address 127.8.204.65:34303)
0806 23:57:04.627318 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:04.627355 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:04.627963 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:04.628400 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:04.628405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:04.628701 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:04.629078 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:04.629089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.629788 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.629790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:04.630597 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:04.630606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.631111 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.631114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:04.631250 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:04.631675 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:04.631689 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:04.633178 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:04.635175 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:04.635180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:04.635192 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:04.635422 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:04.635719 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:04.635722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:04.635723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:04.635875 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:04.636303 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:04.636310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:04.636596 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 23:57:04.636793 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 23:57:04.637090 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":113,"threads_started":1}
I20260806 23:57:04.638985 12723 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45899
I20260806 23:57:04.639226 12723 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:04.639648 12723 heartbeater.cc:511] Master 127.8.204.126:45899 requested a full tablet report, sending...
I20260806 23:57:04.641228 12444 ts_manager.cc:194] Registered new tserver with Master: 9fdc19e9f2504f9e921dc8a74032e46d (127.8.204.65:41103)
I20260806 23:57:04.642765 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:34303
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:04.659229 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:04.759495 12730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:04.759676 12730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:04.759709 12730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:04.763346 12730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:04.763391 12730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:04.763449 12730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:04.767849 12730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:04.768728 12730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:04.769744 12730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:04.778743 12738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:04.778769 12737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.778859 12730 server_base.cc:1034] running on GCE node
W20260806 23:57:04.778892 12740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.779410 12730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:04.779939 12730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:04.781113 12730 hybrid_clock.cc:648] HybridClock initialized: now 1786060624781084 us; error 48 us; skew 500 ppm
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:04.783463 12730 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:04.784255 12730 webserver.cc:460] Webserver started at http://127.8.204.66:37239/ using document root <none> and password file <none>
I20260806 23:57:04.784686 12730 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:04.784736 12730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:04.784871 12730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:04.786474 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "cd658c8a9e6c40e09a7c6f5ffb0b5b28"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "45c82a3270ddaf251d75b034fafad71d"
server_key_iv: "547b5ae0962ed05bfc2c1993384b6bfa"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.786945 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "cd658c8a9e6c40e09a7c6f5ffb0b5b28"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "45c82a3270ddaf251d75b034fafad71d"
server_key_iv: "547b5ae0962ed05bfc2c1993384b6bfa"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.790046 12730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 23:57:04.792322 12750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.793382 12730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 23:57:04.793462 12730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "cd658c8a9e6c40e09a7c6f5ffb0b5b28"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "45c82a3270ddaf251d75b034fafad71d"
server_key_iv: "547b5ae0962ed05bfc2c1993384b6bfa"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.793529 12730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:04.812088 12730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:04.814426 12730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:04.814546 12730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:04.815042 12730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:04.815708 12730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:04.815752 12730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.815784 12730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:04.815798 12730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.823933 12730 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:36861
I20260806 23:57:04.823957 12909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:36861 every 8 connection(s)
I20260806 23:57:04.824661 12730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:04.834549 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 12730
I20260806 23:57:04.834668 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:04.834992 9009 external_mini_cluster.cc:1428] Setting key 6fe200185af7850f375f9a1ed0d0fd37
I20260806 23:57:04.836279 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:04.826137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:42031 (local address 127.8.204.126:45899)
0806 23:57:04.826255 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:04.826258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:04.826965 (+ 707us) server_negotiation.cc:407] Connection header received
0806 23:57:04.827997 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:04.827999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:04.828041 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:04.828106 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:04.829034 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.829469 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.830451 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.830652 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.830781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:04.833091 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:04.833094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:04.833096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:04.833118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:04.834258 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.834807 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.834810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.834811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.834890 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:04.835264 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:04.835268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:04.835269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:04.835461 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:04.835518 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:04.836027 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 23:57:04.836133 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:04.837090 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:04.826421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45899 (local address 127.8.204.66:42031)
0806 23:57:04.826846 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:04.826881 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:04.827836 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:04.828256 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:04.828261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:04.828548 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:04.828924 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:04.828935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.829563 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.829566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:04.830350 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:04.830359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:04.830831 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:04.830834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:04.830961 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:04.831509 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:04.831524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:04.832938 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:04.834369 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:04.834375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:04.834389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:04.834723 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:04.834976 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:04.834979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:04.834980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:04.835147 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:04.835584 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:04.835589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:04.835816 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 23:57:04.835980 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 23:57:04.836271 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260806 23:57:04.838131 12911 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45899
I20260806 23:57:04.838348 12911 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:04.838852 12911 heartbeater.cc:511] Master 127.8.204.126:45899 requested a full tablet report, sending...
I20260806 23:57:04.839850 12444 ts_manager.cc:194] Registered new tserver with Master: cd658c8a9e6c40e09a7c6f5ffb0b5b28 (127.8.204.66:36861)
I20260806 23:57:04.840665 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:42031
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:04.861194 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:04.963171 12917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:04.963387 12917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:04.963428 12917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:04.967171 12917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:04.967216 12917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:04.967309 12917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:04.972361 12917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:04.973266 12917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:04.974152 12917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:04.983150 12924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:04.983165 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:04.983139 12923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:04.983412 12917 server_base.cc:1034] running on GCE node
I20260806 23:57:04.983814 12917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:04.984365 12917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:04.985570 12917 hybrid_clock.cc:648] HybridClock initialized: now 1786060624985554 us; error 35 us; skew 500 ppm
Aug 06 23:57:04 dist-test-slave-39l8 krb5kdc[12383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:04.988142 12917 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:04.988911 12917 webserver.cc:460] Webserver started at http://127.8.204.67:33371/ using document root <none> and password file <none>
I20260806 23:57:04.989467 12917 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:04.989574 12917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:04.989786 12917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:04.991811 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "c9916d04a882409d82dcfa997e2e1762"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "6cddce20cc98a16cb1383328e654499b"
server_key_iv: "13cd0e35df893c1a6cf558b0d81eef26"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.992358 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "c9916d04a882409d82dcfa997e2e1762"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "6cddce20cc98a16cb1383328e654499b"
server_key_iv: "13cd0e35df893c1a6cf558b0d81eef26"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.995620 12917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:04.997906 12934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:04.999022 12917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:04.999130 12917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "c9916d04a882409d82dcfa997e2e1762"
format_stamp: "Formatted at 2026-08-06 23:57:04 on dist-test-slave-39l8"
server_key: "6cddce20cc98a16cb1383328e654499b"
server_key_iv: "13cd0e35df893c1a6cf558b0d81eef26"
server_key_version: "encryptionkey@0"
I20260806 23:57:04.999217 12917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:05.021591 12917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:05.024040 12917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:05.024178 12917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:05.024645 12917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:05.025331 12917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:05.025373 12917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:05.025408 12917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:05.025422 12917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:05.034188 12917 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:45539
I20260806 23:57:05.034240 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:45539 every 8 connection(s)
I20260806 23:57:05.034993 12917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:05.036659 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 12917
I20260806 23:57:05.036773 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:05.037076 9009 external_mini_cluster.cc:1428] Setting key 46f7e40ae6b28b469b121902cc7e63b1
Aug 06 23:57:05 dist-test-slave-39l8 krb5kdc[12383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:05.047008 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:05.036789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:43177 (local address 127.8.204.126:45899)
0806 23:57:05.036927 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:05.036931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:05.038068 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 23:57:05.038954 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:05.038957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:05.039027 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:05.039133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:05.040149 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:05.040688 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:05.041738 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:05.041921 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:05.041992 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:05.044044 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:05.044047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:05.044049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:05.044070 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:05.045458 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:05.045858 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:05.045861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:05.045862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:05.045928 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:05.046187 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:05.046190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:05.046191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:05.046331 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:05.046402 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:05.046818 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 23:57:05.046904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:57:05.047680 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:05.037256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45899 (local address 127.8.204.67:43177)
0806 23:57:05.037933 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:05.037980 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:05.038674 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:05.039315 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:05.039322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:05.039631 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:05.040026 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:05.040037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:05.040794 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:05.040797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:05.041636 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:05.041644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:05.042035 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:05.042038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:05.042167 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:05.042587 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:05.042601 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:05.043892 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:05.045560 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:05.045564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:05.045573 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:05.045780 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:05.046001 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:05.046002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:05.046003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:05.046114 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:05.046413 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:05.046417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:05.046593 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 23:57:05.046779 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 23:57:05.046983 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":138,"threads_started":1}
I20260806 23:57:05.048813 13102 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45899
I20260806 23:57:05.048997 13102 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:05.049384 13102 heartbeater.cc:511] Master 127.8.204.126:45899 requested a full tablet report, sending...
I20260806 23:57:05.050500 12444 ts_manager.cc:194] Registered new tserver with Master: c9916d04a882409d82dcfa997e2e1762 (127.8.204.67:45539)
I20260806 23:57:05.051116 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:57:05.051275 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:43177
W20260806 23:57:05.066249 12501 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:05.064937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.8.204.126:45899)
0806 23:57:05.065081 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:05.065084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:05.065269 (+ 185us) server_negotiation.cc:407] Connection header received
0806 23:57:05.065366 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:05.065368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:05.065404 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:05.065474 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:05.066110 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50058: BlockingRecv error: recv got EOF from 127.0.0.1:50058 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 23:57:05.067293 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 12536
I20260806 23:57:05.080880 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 12730
I20260806 23:57:05.091652 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 12917
I20260806 23:57:05.100445 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 12400
2026-08-06T23:57:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2194 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:05 dist-test-slave-39l8 krb5kdc[13121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:05 dist-test-slave-39l8 krb5kdc[13121](info): set up 2 sockets
Aug 06 23:57:05 dist-test-slave-39l8 krb5kdc[13121](info): commencing operation
krb5kdc: starting...
W20260806 23:57:06.238835 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:06.349596 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35269
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:06.453033 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:06.453255 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:06.453308 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:06.457167 13139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:06.457217 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:06.457237 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:06.457252 13139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:06.457266 13139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:06.462029 13139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:06.462889 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:06.463730 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:06.471487 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:06.471493 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:06.471513 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:06.471797 13139 server_base.cc:1034] running on GCE node
I20260806 23:57:06.472365 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:06.473348 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:06.474536 13139 hybrid_clock.cc:648] HybridClock initialized: now 1786060626474508 us; error 41 us; skew 500 ppm
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060626, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:06.477201 13139 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:06.478050 13139 webserver.cc:460] Webserver started at http://127.8.204.126:39001/ using document root <none> and password file <none>
I20260806 23:57:06.478581 13139 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:06.478665 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:06.478904 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:06.480762 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "7482bf0cefb14c359df51725fe80dbd5"
format_stamp: "Formatted at 2026-08-06 23:57:06 on dist-test-slave-39l8"
server_key: "64741b71e41be14d29cebbf88fa0af52"
server_key_iv: "101d8a193c53f2e049672719e31cd883"
server_key_version: "encryptionkey@0"
I20260806 23:57:06.481312 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "7482bf0cefb14c359df51725fe80dbd5"
format_stamp: "Formatted at 2026-08-06 23:57:06 on dist-test-slave-39l8"
server_key: "64741b71e41be14d29cebbf88fa0af52"
server_key_iv: "101d8a193c53f2e049672719e31cd883"
server_key_version: "encryptionkey@0"
I20260806 23:57:06.484704 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:06.486912 13167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:06.488003 13139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:06.488148 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "7482bf0cefb14c359df51725fe80dbd5"
format_stamp: "Formatted at 2026-08-06 23:57:06 on dist-test-slave-39l8"
server_key: "64741b71e41be14d29cebbf88fa0af52"
server_key_iv: "101d8a193c53f2e049672719e31cd883"
server_key_version: "encryptionkey@0"
I20260806 23:57:06.488245 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:06.499207 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:06.502152 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:06.502324 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:06.508877 13139 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:35269
I20260806 23:57:06.509156 13248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:35269 every 8 connection(s)
I20260806 23:57:06.509737 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:06.512390 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:06.514851 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 13139
I20260806 23:57:06.514943 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:06.515213 9009 external_mini_cluster.cc:1428] Setting key 4e5e315bce31cb6703e491d2a58a8578
I20260806 23:57:06.518333 13250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5: Bootstrap starting.
I20260806 23:57:06.520979 13250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:06.521754 13250 log.cc:824] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:06.523680 13250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5: No bootstrap required, opened a new log
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:06.527459 13250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } }
I20260806 23:57:06.527750 13250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:06.527832 13250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7482bf0cefb14c359df51725fe80dbd5, State: Initialized, Role: FOLLOWER
I20260806 23:57:06.528337 13250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } }
I20260806 23:57:06.528469 13250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:06.528550 13250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:06.528649 13250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:06.529594 13250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } }
I20260806 23:57:06.529935 13250 leader_election.cc:302] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7482bf0cefb14c359df51725fe80dbd5; no voters:
I20260806 23:57:06.530289 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:06.516522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.8.204.126:35269)
0806 23:57:06.516964 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:06.516981 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:06.517024 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:57:06.517817 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:06.517838 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:06.518240 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:06.518580 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:06.519326 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:06.520806 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:06.521721 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:06.522313 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:06.522463 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:06.525098 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:06.525102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:06.525114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:06.525139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:06.527198 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:06.527686 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:06.527690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:06.527694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:06.527887 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:06.528222 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:06.528224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:06.528225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:06.528682 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:06.528925 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:06.529187 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 23:57:06.529463 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260806 23:57:06.530335 13257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:06.530287 13250 leader_election.cc:288] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:06.530587 13257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [term 1 LEADER]: Becoming Leader. State: Replica: 7482bf0cefb14c359df51725fe80dbd5, State: Running, Role: LEADER
I20260806 23:57:06.530916 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } }
I20260806 23:57:06.531664 13250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:06.532111 13257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7482bf0cefb14c359df51725fe80dbd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } } }
I20260806 23:57:06.532310 13257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:06.532896 13262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:06.533452 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:57:06.534511 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7482bf0cefb14c359df51725fe80dbd5. Latest consensus state: current_term: 1 leader_uuid: "7482bf0cefb14c359df51725fe80dbd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7482bf0cefb14c359df51725fe80dbd5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35269 } } }
I20260806 23:57:06.534595 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:06.536861 13262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:06.542830 13262 catalog_manager.cc:1347] Generated new cluster ID: 2f48d95e85fd4e5095429f06cff8b8d9
I20260806 23:57:06.542936 13262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:06.558089 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:06.552718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.8.204.126:35269)
0806 23:57:06.552829 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:06.552835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:06.553074 (+ 239us) server_negotiation.cc:407] Connection header received
0806 23:57:06.553169 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:06.553172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:06.553220 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:06.553312 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:06.553842 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:06.554610 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:06.555706 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:06.556060 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:06.556165 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:06.557053 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:06.557057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:06.557061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:06.557620 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:06.557655 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:06.557674 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:06.557764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:06.557860 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 23:57:06.557915 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:57:06.575337 13262 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:06.576524 13262 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:06.582089 13262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7482bf0cefb14c359df51725fe80dbd5: Generated new TSK 0
I20260806 23:57:06.582804 13262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:06.593055 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 13139
2026-08-06T23:57:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1495 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13297](info): set up 2 sockets
Aug 06 23:57:06 dist-test-slave-39l8 krb5kdc[13297](info): commencing operation
krb5kdc: starting...
W20260806 23:57:07.763659 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:07 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:07.878551 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37099
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:07.982009 13315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:07.982255 13315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:07.982354 13315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:07.985908 13315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:07.985984 13315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:07.986018 13315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:07.986066 13315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:07.986100 13315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:07.990774 13315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:07.991678 13315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:07.992610 13315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:08.000115 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:08.000154 13315 server_base.cc:1034] running on GCE node
W20260806 23:57:08.000113 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.000120 13328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:08.000887 13315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:08.001874 13315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:08.003049 13315 hybrid_clock.cc:648] HybridClock initialized: now 1786060628003025 us; error 36 us; skew 500 ppm
Aug 06 23:57:07 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060627, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:08.005573 13315 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:08.006412 13315 webserver.cc:460] Webserver started at http://127.8.204.126:39117/ using document root <none> and password file <none>
I20260806 23:57:08.006985 13315 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:08.007076 13315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:08.007323 13315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:08.009166 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "7ebd9ff4493744358e276cf95b4eaecf"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "77306d923ae3b977a1cde280796860f1"
server_key_iv: "773b9cd2fb0bb079988cfe4fc45e4b11"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.009703 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "7ebd9ff4493744358e276cf95b4eaecf"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "77306d923ae3b977a1cde280796860f1"
server_key_iv: "773b9cd2fb0bb079988cfe4fc45e4b11"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.014998 13315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 23:57:08.017802 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.019217 13315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:57:08.019354 13315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "7ebd9ff4493744358e276cf95b4eaecf"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "77306d923ae3b977a1cde280796860f1"
server_key_iv: "773b9cd2fb0bb079988cfe4fc45e4b11"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.019455 13315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:08.033231 13315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:08.036175 13315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:08.036341 13315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:08.042909 13315 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37099
I20260806 23:57:08.043171 13412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37099 every 8 connection(s)
I20260806 23:57:08.043812 13315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:08.044150 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 13315
I20260806 23:57:08.044273 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:08.044565 9009 external_mini_cluster.cc:1428] Setting key 5d1a47b810c9935d8be7c8aa53424adb
I20260806 23:57:08.046492 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:08.053172 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf: Bootstrap starting.
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:08.055444 13414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:08.056282 13414 log.cc:824] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:08.058127 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf: No bootstrap required, opened a new log
I20260806 23:57:08.058281 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.045736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.8.204.126:37099)
0806 23:57:08.046106 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:08.046120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:08.046151 (+ 31us) server_negotiation.cc:407] Connection header received
0806 23:57:08.046799 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:08.046817 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:08.047093 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:08.047329 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:08.048165 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.049384 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.050557 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.051069 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.051193 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.053935 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:08.053940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:08.053950 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:08.053975 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:08.055659 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.056099 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.056103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.056107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.056206 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.056493 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.056496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.056497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.056950 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:08.057122 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:08.057390 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 23:57:08.057642 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
I20260806 23:57:08.061358 13414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } }
I20260806 23:57:08.061587 13414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:08.061667 13414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebd9ff4493744358e276cf95b4eaecf, State: Initialized, Role: FOLLOWER
I20260806 23:57:08.062071 13414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } }
I20260806 23:57:08.062211 13414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:08.062290 13414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:08.062404 13414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:08.063382 13414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } }
I20260806 23:57:08.063700 13414 leader_election.cc:302] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ebd9ff4493744358e276cf95b4eaecf; no voters:
I20260806 23:57:08.064045 13414 leader_election.cc:288] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:08.064154 13420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:08.064381 13420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [term 1 LEADER]: Becoming Leader. State: Replica: 7ebd9ff4493744358e276cf95b4eaecf, State: Running, Role: LEADER
I20260806 23:57:08.064683 13420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } }
I20260806 23:57:08.065182 13414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:08.066247 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ebd9ff4493744358e276cf95b4eaecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } } }
I20260806 23:57:08.066397 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:08.066538 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebd9ff4493744358e276cf95b4eaecf. Latest consensus state: current_term: 1 leader_uuid: "7ebd9ff4493744358e276cf95b4eaecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebd9ff4493744358e276cf95b4eaecf" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37099 } } }
I20260806 23:57:08.066743 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:08.067543 13432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:08.070061 13432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:08.075425 13432 catalog_manager.cc:1347] Generated new cluster ID: 4f19d10a1a1f4a9889bba7ef1104dc5f
I20260806 23:57:08.075484 13432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:08.103374 13432 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:08.104588 13432 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:08.116493 13432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ebd9ff4493744358e276cf95b4eaecf: Generated new TSK 0
I20260806 23:57:08.117264 13432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:08.147607 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:08.249850 13446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:08.250041 13446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:08.250074 13446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:08.253863 13446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:08.253907 13446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:08.253955 13446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:08.258294 13446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:08.259158 13446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:08.260006 13446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:08.269223 13455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.269183 13457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.269205 13454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:08.269265 13446 server_base.cc:1034] running on GCE node
I20260806 23:57:08.269856 13446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:08.270421 13446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:08.271607 13446 hybrid_clock.cc:648] HybridClock initialized: now 1786060628271562 us; error 59 us; skew 500 ppm
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:08.274183 13446 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:08.275087 13446 webserver.cc:460] Webserver started at http://127.8.204.65:45139/ using document root <none> and password file <none>
I20260806 23:57:08.275600 13446 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:08.275688 13446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:08.275904 13446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:08.277797 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "91bb71b976904f5daca1900d18b9e0c4"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "2ef50e9155feeedb354e6e1926a4fdc8"
server_key_iv: "f47e29a2e8b40edc05524de26ce07666"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.278337 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "91bb71b976904f5daca1900d18b9e0c4"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "2ef50e9155feeedb354e6e1926a4fdc8"
server_key_iv: "f47e29a2e8b40edc05524de26ce07666"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.281661 13446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:57:08.284149 13467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.285197 13446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:08.285347 13446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "91bb71b976904f5daca1900d18b9e0c4"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "2ef50e9155feeedb354e6e1926a4fdc8"
server_key_iv: "f47e29a2e8b40edc05524de26ce07666"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.285465 13446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:08.300351 13446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:08.302831 13446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:08.303002 13446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:08.303508 13446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:08.304278 13446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:08.304378 13446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.304463 13446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:08.304514 13446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.312649 13446 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:43787
I20260806 23:57:08.312685 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:43787 every 8 connection(s)
I20260806 23:57:08.313421 13446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:08.322667 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 13446
I20260806 23:57:08.322815 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:08.323120 9009 external_mini_cluster.cc:1428] Setting key 04df24bb7fd4c4f11f6444330c8ed7e2
I20260806 23:57:08.325501 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.314990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35739 (local address 127.8.204.126:37099)
0806 23:57:08.315156 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:08.315161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:08.315851 (+ 690us) server_negotiation.cc:407] Connection header received
0806 23:57:08.316966 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:08.316968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:08.317015 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:08.317094 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:08.318059 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.318577 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.319600 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.319826 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.319903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.322058 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:08.322062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:08.322064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:08.322098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:08.323528 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.324072 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.324075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.324077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.324155 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.324551 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.324553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.324554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.324715 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:08.324789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:08.325317 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 23:57:08.325401 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 23:57:08.326521 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.315302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37099 (local address 127.8.204.65:35739)
0806 23:57:08.315684 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:08.315734 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:08.316791 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:08.317232 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:08.317238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:08.317536 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:08.317933 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.317943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.318678 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.318680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.319499 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.319508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.319949 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.319952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.320073 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.320518 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:08.320550 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:08.321900 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:08.323663 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.323669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.323699 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.323977 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.324261 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.324264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.324265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.324452 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.324826 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:08.324832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:08.325106 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 23:57:08.325285 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 23:57:08.325586 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260806 23:57:08.327592 13645 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37099
I20260806 23:57:08.327814 13645 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:08.328210 13645 heartbeater.cc:511] Master 127.8.204.126:37099 requested a full tablet report, sending...
I20260806 23:57:08.329656 13359 ts_manager.cc:194] Registered new tserver with Master: 91bb71b976904f5daca1900d18b9e0c4 (127.8.204.65:43787)
I20260806 23:57:08.331302 13359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:35739
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:08.349639 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:08.452828 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:08.453094 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:08.453171 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:08.456712 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:08.456781 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:08.456853 13655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:08.461522 13655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:08.462414 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:08.463299 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:08.472448 13662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.472486 13663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.472702 13665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:08.472479 13655 server_base.cc:1034] running on GCE node
I20260806 23:57:08.473129 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:08.473728 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:08.474901 13655 hybrid_clock.cc:648] HybridClock initialized: now 1786060628474881 us; error 39 us; skew 500 ppm
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:08.477427 13655 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:08.478242 13655 webserver.cc:460] Webserver started at http://127.8.204.66:35917/ using document root <none> and password file <none>
I20260806 23:57:08.478827 13655 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:08.478914 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:08.479141 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:08.481025 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "a800d18696114a2b91d321b2b03b330e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "8050de4191f8d191413b4e95cd64b6e2"
server_key_iv: "aa6c1337dcac56932a0021a2a4aa4a40"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.481585 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "a800d18696114a2b91d321b2b03b330e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "8050de4191f8d191413b4e95cd64b6e2"
server_key_iv: "aa6c1337dcac56932a0021a2a4aa4a40"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.484979 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:08.487134 13675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.488219 13655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:08.488348 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "a800d18696114a2b91d321b2b03b330e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "8050de4191f8d191413b4e95cd64b6e2"
server_key_iv: "aa6c1337dcac56932a0021a2a4aa4a40"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.488448 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:08.503365 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:08.505864 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:08.506054 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:08.506580 13655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:08.507385 13655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:08.507457 13655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.507520 13655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:08.507557 13655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.516037 13655 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44411
I20260806 23:57:08.516096 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44411 every 8 connection(s)
I20260806 23:57:08.516827 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:08.525223 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 13655
I20260806 23:57:08.525388 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:08.525673 9009 external_mini_cluster.cc:1428] Setting key aa7af46bbbd2fbbb6b1164bfe74e9cc8
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:08.529992 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.518404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:47603 (local address 127.8.204.126:37099)
0806 23:57:08.518526 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:08.518530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:08.519276 (+ 746us) server_negotiation.cc:407] Connection header received
0806 23:57:08.520569 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:08.520573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:08.520652 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:08.520731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:08.521880 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.522447 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.523641 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.523862 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.523973 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.526339 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:08.526342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:08.526344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:08.526393 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:08.527798 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.528376 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.528378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.528379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.528447 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.528811 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.528813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.528814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.528989 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:08.529073 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:08.529658 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 23:57:08.529790 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:57:08.530541 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.518666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37099 (local address 127.8.204.66:47603)
0806 23:57:08.519139 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:08.519176 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:08.520362 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:08.520918 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:08.520932 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:08.521260 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:08.521675 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.521717 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.522595 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.522602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.523520 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.523529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.524007 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.524011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.524160 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.524696 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:08.524711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:08.526140 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:08.527913 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.527921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.527936 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.528279 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.528533 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.528539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.528543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.528723 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.529091 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:08.529100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:08.529402 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 23:57:08.529572 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 23:57:08.529797 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":106,"threads_started":1}
I20260806 23:57:08.531662 13835 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37099
I20260806 23:57:08.531906 13835 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:08.532370 13835 heartbeater.cc:511] Master 127.8.204.126:37099 requested a full tablet report, sending...
I20260806 23:57:08.533406 13359 ts_manager.cc:194] Registered new tserver with Master: a800d18696114a2b91d321b2b03b330e (127.8.204.66:44411)
I20260806 23:57:08.533938 13359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:47603
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:08.552827 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:08.659495 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:08.659700 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:08.659754 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:08.663378 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:08.663420 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:08.663470 13841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:08.668382 13841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:08.669332 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:08.670266 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:08.679329 13851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.679399 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:08.679329 13855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:08.679415 13841 server_base.cc:1034] running on GCE node
I20260806 23:57:08.679980 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:08.680591 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:08.681770 13841 hybrid_clock.cc:648] HybridClock initialized: now 1786060628681753 us; error 29 us; skew 500 ppm
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:08.684583 13841 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:08.685389 13841 webserver.cc:460] Webserver started at http://127.8.204.67:37839/ using document root <none> and password file <none>
I20260806 23:57:08.685935 13841 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:08.686024 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:08.686272 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:08.688547 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "950f0b3145104c76b184a2dfa67af07e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "91a1b17cc162509a2a2585a4189e7c78"
server_key_iv: "4ef8222fa4bb711e0e41a1d74f4df050"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.689102 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "950f0b3145104c76b184a2dfa67af07e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "91a1b17cc162509a2a2585a4189e7c78"
server_key_iv: "4ef8222fa4bb711e0e41a1d74f4df050"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.692924 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 23:57:08.695289 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.696456 13841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:08.696547 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "950f0b3145104c76b184a2dfa67af07e"
format_stamp: "Formatted at 2026-08-06 23:57:08 on dist-test-slave-39l8"
server_key: "91a1b17cc162509a2a2585a4189e7c78"
server_key_iv: "4ef8222fa4bb711e0e41a1d74f4df050"
server_key_version: "encryptionkey@0"
I20260806 23:57:08.696622 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:08.717809 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:08.720959 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:08.721107 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:08.721638 13841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:08.722522 13841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:08.722566 13841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.722646 13841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:08.722688 13841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:08.731592 13841 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:42949
I20260806 23:57:08.731630 14026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:42949 every 8 connection(s)
I20260806 23:57:08.732477 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:08.738662 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 13841
I20260806 23:57:08.738790 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:08.739070 9009 external_mini_cluster.cc:1428] Setting key bb8b9b56eb487ab0000faf8e32b45652
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:08.745434 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.734283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:38017 (local address 127.8.204.126:37099)
0806 23:57:08.734420 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:08.734423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:08.735253 (+ 830us) server_negotiation.cc:407] Connection header received
0806 23:57:08.736464 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:08.736467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:08.736522 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:08.736624 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:08.737644 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.738124 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.739302 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.739594 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.739700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.742290 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:08.742294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:08.742296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:08.742321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:08.743578 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.744038 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.744042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.744044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.744129 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.744463 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.744467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.744469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.744636 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:08.744741 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:08.745170 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 23:57:08.745280 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:57:08.746064 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.734640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37099 (local address 127.8.204.67:38017)
0806 23:57:08.735113 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:08.735150 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:08.736255 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:08.736784 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:08.736790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:08.737095 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:08.737497 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.737507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.738240 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.738243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.739175 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:08.739187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.739748 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.739752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:08.739971 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.740508 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:08.740530 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:08.742124 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:08.743685 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.743691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.743702 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.743943 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.744213 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:08.744217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:08.744218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:08.744368 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:08.744739 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:08.744745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:08.744956 (+ 211us) client_negotiation.cc:769] Sending connection context
0806 23:57:08.745126 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 23:57:08.745392 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":112,"threads_started":1}
I20260806 23:57:08.747048 14027 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37099
I20260806 23:57:08.747257 14027 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:08.747695 14027 heartbeater.cc:511] Master 127.8.204.126:37099 requested a full tablet report, sending...
I20260806 23:57:08.748595 13359 ts_manager.cc:194] Registered new tserver with Master: 950f0b3145104c76b184a2dfa67af07e (127.8.204.67:42949)
I20260806 23:57:08.749105 13359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:38017
I20260806 23:57:08.752674 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[13297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060628, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:08.777591 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:08.770076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.8.204.126:37099)
0806 23:57:08.770205 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:08.770209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:08.770408 (+ 199us) server_negotiation.cc:407] Connection header received
0806 23:57:08.770506 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:08.770508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:08.770557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:08.770650 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:08.771358 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.771852 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.772632 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:08.772846 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:08.772922 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:08.775169 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:08.775172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:08.775173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:08.775192 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:08.776285 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.776624 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.776626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.776627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.776692 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:08.776899 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:08.776901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:08.776902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:08.777114 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:08.777214 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:08.777400 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 23:57:08.777485 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 23:57:08.777946 13359 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54154
I20260806 23:57:08.778875 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 13446
I20260806 23:57:08.791841 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 13655
I20260806 23:57:08.801254 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 13841
I20260806 23:57:08.809408 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 13315
2026-08-06T23:57:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2213 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[14044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[14044](info): set up 2 sockets
Aug 06 23:57:08 dist-test-slave-39l8 krb5kdc[14044](info): commencing operation
krb5kdc: starting...
W20260806 23:57:09.962401 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:10.081329 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40279
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:10.186908 14061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:10.187158 14061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:10.187227 14061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:10.191053 14061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:10.191118 14061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:10.191155 14061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:10.191207 14061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:10.191247 14061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:10.196033 14061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:10.196970 14061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:10.197858 14061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:10.205519 14071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.205552 14076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.205554 14061 server_base.cc:1034] running on GCE node
W20260806 23:57:10.205770 14080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.206269 14061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:10.207326 14061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:10.208514 14061 hybrid_clock.cc:648] HybridClock initialized: now 1786060630208492 us; error 36 us; skew 500 ppm
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:10.217567 14061 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:10.218356 14061 webserver.cc:460] Webserver started at http://127.8.204.126:42405/ using document root <none> and password file <none>
I20260806 23:57:10.218904 14061 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:10.218961 14061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:10.219111 14061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:10.220892 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "03fc8cf42ba149d79e6e9a03fb062ec5"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "8840f72a6ac65ca01af42e933f442a17"
server_key_iv: "1aabb3f1c2399927cc062da96cb082c0"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.221393 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "03fc8cf42ba149d79e6e9a03fb062ec5"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "8840f72a6ac65ca01af42e933f442a17"
server_key_iv: "1aabb3f1c2399927cc062da96cb082c0"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.224709 14061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:10.226882 14090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.227960 14061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:10.228075 14061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "03fc8cf42ba149d79e6e9a03fb062ec5"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "8840f72a6ac65ca01af42e933f442a17"
server_key_iv: "1aabb3f1c2399927cc062da96cb082c0"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.228164 14061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:10.245502 14061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:10.248193 14061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:10.248339 14061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:10.254571 14061 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40279
I20260806 23:57:10.254655 14158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40279 every 8 connection(s)
I20260806 23:57:10.255434 14061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:10.257565 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14061
I20260806 23:57:10.257684 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:10.257972 9009 external_mini_cluster.cc:1428] Setting key a26add0040ec768a30de04b9156e003d
I20260806 23:57:10.258188 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:10.265254 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: Bootstrap starting.
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:10.268005 14160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:10.268951 14160 log.cc:824] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:10.270865 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: No bootstrap required, opened a new log
I20260806 23:57:10.272557 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.259269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.8.204.126:40279)
0806 23:57:10.259709 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.259728 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.259767 (+ 39us) server_negotiation.cc:407] Connection header received
0806 23:57:10.260510 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.260533 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.260942 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.261308 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.261970 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.263339 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.264100 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.264821 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.264991 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.267593 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.267599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.267614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.267645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.269599 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.270034 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.270039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.270044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.270158 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.270436 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.270439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.270441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.270911 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.271100 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.271394 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.271675 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260806 23:57:10.274120 14160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } }
I20260806 23:57:10.274370 14160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:10.274439 14160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fc8cf42ba149d79e6e9a03fb062ec5, State: Initialized, Role: FOLLOWER
I20260806 23:57:10.274937 14160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } }
I20260806 23:57:10.275061 14160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:10.275137 14160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:10.275238 14160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:10.276060 14160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } }
I20260806 23:57:10.276396 14160 leader_election.cc:302] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03fc8cf42ba149d79e6e9a03fb062ec5; no voters:
I20260806 23:57:10.276739 14160 leader_election.cc:288] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:10.276811 14168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:10.277064 14168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 03fc8cf42ba149d79e6e9a03fb062ec5, State: Running, Role: LEADER
I20260806 23:57:10.277402 14168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } }
I20260806 23:57:10.277858 14160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:10.278910 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } } }
I20260806 23:57:10.279048 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:10.279342 14171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03fc8cf42ba149d79e6e9a03fb062ec5. Latest consensus state: current_term: 1 leader_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fc8cf42ba149d79e6e9a03fb062ec5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40279 } } }
I20260806 23:57:10.279430 14171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5 [sys.catalog]: This master's current role is: LEADER
W20260806 23:57:10.282524 14185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:57:10.282588 14185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:57:10.282738 14179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:10.283476 14179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:10.288578 14179 catalog_manager.cc:1347] Generated new cluster ID: 89fdb639aa8c48c68cb3c6f38f6d0588
I20260806 23:57:10.288633 14179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:10.298002 14179 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:10.299158 14179 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:10.308243 14179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03fc8cf42ba149d79e6e9a03fb062ec5: Generated new TSK 0
I20260806 23:57:10.308913 14179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:10.335176 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:10.443423 14191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:10.443677 14191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:10.443747 14191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:10.447795 14191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:10.447865 14191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:10.447944 14191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:10.453279 14191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:10.454250 14191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:10.455276 14191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:10.464172 14201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.464205 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.464205 14199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.464435 14191 server_base.cc:1034] running on GCE node
I20260806 23:57:10.464856 14191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:10.465472 14191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:10.466681 14191 hybrid_clock.cc:648] HybridClock initialized: now 1786060630466666 us; error 23 us; skew 500 ppm
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:10.469271 14191 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:10.470155 14191 webserver.cc:460] Webserver started at http://127.8.204.65:39101/ using document root <none> and password file <none>
I20260806 23:57:10.470795 14191 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:10.470889 14191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:10.471129 14191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:10.473596 14191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "69134695f89b4bf398204f4a3b456055"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e20e49267e48a990b5e2c5bd9d3577e0"
server_key_iv: "0aa7f8454e302bfde215e7827c46cd32"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.474138 14191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "69134695f89b4bf398204f4a3b456055"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e20e49267e48a990b5e2c5bd9d3577e0"
server_key_iv: "0aa7f8454e302bfde215e7827c46cd32"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.477663 14191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 23:57:10.480297 14217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.481472 14191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:10.481621 14191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "69134695f89b4bf398204f4a3b456055"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e20e49267e48a990b5e2c5bd9d3577e0"
server_key_iv: "0aa7f8454e302bfde215e7827c46cd32"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.481734 14191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:10.493743 14191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:10.496426 14191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:10.496630 14191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:10.497223 14191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:10.497990 14191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:10.498065 14191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.498134 14191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:10.498175 14191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.507217 14191 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:39983
I20260806 23:57:10.507244 14373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:39983 every 8 connection(s)
I20260806 23:57:10.508138 14191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:10.511009 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14191
I20260806 23:57:10.511138 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:10.511487 9009 external_mini_cluster.cc:1428] Setting key c824630c546283ba9fc8ef97b71f5dca
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:10.522645 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.509846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:59831 (local address 127.8.204.126:40279)
0806 23:57:10.509992 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.509997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.510901 (+ 904us) server_negotiation.cc:407] Connection header received
0806 23:57:10.512614 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.512617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.512659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.512728 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.514174 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.514671 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.516010 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.516427 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.516555 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.518904 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.518911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.518917 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.518951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.520587 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.521071 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.521077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.521081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.521178 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.521496 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.521500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.521504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.521751 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.521857 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.522323 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.522467 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:57:10.523278 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.510105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40279 (local address 127.8.204.65:59831)
0806 23:57:10.510666 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:10.510785 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:10.512392 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:10.512926 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:10.512933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:10.513383 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:10.514011 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.514034 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.514856 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.514863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.515885 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.515896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.516568 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.516570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.516718 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.517261 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:10.517276 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:10.518706 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:10.520706 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.520711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.520721 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.520968 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.521275 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.521277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.521279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.521403 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.521866 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:10.521884 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:10.522086 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 23:57:10.522228 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 23:57:10.522450 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":100,"threads_started":1}
I20260806 23:57:10.524325 14374 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40279
I20260806 23:57:10.524560 14374 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:10.525044 14374 heartbeater.cc:511] Master 127.8.204.126:40279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260806 23:57:10.526821 14116 ts_manager.cc:194] Registered new tserver with Master: 69134695f89b4bf398204f4a3b456055 (127.8.204.65:39983)
I20260806 23:57:10.527912 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:59831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:10.539680 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:10.643576 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:10.643843 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:10.643913 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:10.647701 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:10.647775 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:10.647855 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:10.652773 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:10.653741 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:10.654661 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:10.663928 14395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.663928 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.663934 14393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.663981 14385 server_base.cc:1034] running on GCE node
I20260806 23:57:10.664482 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:10.665052 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:10.666239 14385 hybrid_clock.cc:648] HybridClock initialized: now 1786060630666211 us; error 39 us; skew 500 ppm
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:10.668780 14385 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:10.669602 14385 webserver.cc:460] Webserver started at http://127.8.204.66:38093/ using document root <none> and password file <none>
I20260806 23:57:10.670141 14385 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:10.670230 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:10.670482 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:10.672534 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "a19a06365d0a4051af62f6c470df05e7"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e154ab4e8753fe1399d93a04f38958a7"
server_key_iv: "687a02581f441f09da5db24273ab3526"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.673085 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "a19a06365d0a4051af62f6c470df05e7"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e154ab4e8753fe1399d93a04f38958a7"
server_key_iv: "687a02581f441f09da5db24273ab3526"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.676477 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:10.678839 14410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.679944 14385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:10.680063 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "a19a06365d0a4051af62f6c470df05e7"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "e154ab4e8753fe1399d93a04f38958a7"
server_key_iv: "687a02581f441f09da5db24273ab3526"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.680184 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:10.702548 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:10.705364 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:10.705564 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:10.706142 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:10.707008 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:10.707072 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.707134 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:10.707170 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.715615 14385 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:35529
I20260806 23:57:10.715646 14571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:35529 every 8 connection(s)
I20260806 23:57:10.716475 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:10.725416 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14385
I20260806 23:57:10.725539 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:10.725871 9009 external_mini_cluster.cc:1428] Setting key cb7e8164ad79d439b3f3102ed9a3728d
I20260806 23:57:10.729482 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.718206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48291 (local address 127.8.204.126:40279)
0806 23:57:10.718345 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.718349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.719266 (+ 917us) server_negotiation.cc:407] Connection header received
0806 23:57:10.720562 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.720564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.720608 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.720677 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.721793 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.722299 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.723350 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.723544 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.723620 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.725870 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.725874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.725876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.725909 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.727427 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.727911 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.727917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.727921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.728013 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.728332 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.728338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.728342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.728585 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.728685 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.729124 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.729256 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:57:10.730485 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.718548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40279 (local address 127.8.204.66:48291)
0806 23:57:10.719092 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:10.719148 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:10.720387 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:10.720868 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:10.720874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:10.721211 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:10.721668 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.721680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.722393 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.722395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.723247 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.723257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.723667 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.723669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.723792 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.724254 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:10.724268 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:10.725695 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:10.727549 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.727554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.727564 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.727811 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.728108 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.728110 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.728111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.728242 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.728690 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:10.728694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:10.728903 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 23:57:10.729048 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 23:57:10.729274 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":176,"threads_started":1}
I20260806 23:57:10.731699 14572 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40279
I20260806 23:57:10.731947 14572 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:10.732484 14572 heartbeater.cc:511] Master 127.8.204.126:40279 requested a full tablet report, sending...
I20260806 23:57:10.733496 14116 ts_manager.cc:194] Registered new tserver with Master: a19a06365d0a4051af62f6c470df05e7 (127.8.204.66:35529)
I20260806 23:57:10.734037 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:48291
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:10.753328 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:10.862830 14580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:10.863042 14580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:10.863080 14580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:10.866823 14580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:10.866873 14580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:10.866926 14580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:10.872025 14580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:10.872938 14580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:10.873857 14580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:10.882897 14587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:10.882908 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.883033 14580 server_base.cc:1034] running on GCE node
W20260806 23:57:10.882916 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:10.883670 14580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:10.884308 14580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:10.885473 14580 hybrid_clock.cc:648] HybridClock initialized: now 1786060630885438 us; error 46 us; skew 500 ppm
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:10.888077 14580 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:10.888918 14580 webserver.cc:460] Webserver started at http://127.8.204.67:41225/ using document root <none> and password file <none>
I20260806 23:57:10.889437 14580 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:10.889516 14580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:10.889741 14580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:10.891656 14580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "7c2587a9fe344465937b97e2c01d0fbc"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "01366a8bdb7dc0c8317c3eb250df6529"
server_key_iv: "a23a837b56fb9ed4ce0632636cbee0f1"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.892227 14580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "7c2587a9fe344465937b97e2c01d0fbc"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "01366a8bdb7dc0c8317c3eb250df6529"
server_key_iv: "a23a837b56fb9ed4ce0632636cbee0f1"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.895540 14580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:10.897756 14600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.898851 14580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:10.898965 14580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "7c2587a9fe344465937b97e2c01d0fbc"
format_stamp: "Formatted at 2026-08-06 23:57:10 on dist-test-slave-39l8"
server_key: "01366a8bdb7dc0c8317c3eb250df6529"
server_key_iv: "a23a837b56fb9ed4ce0632636cbee0f1"
server_key_version: "encryptionkey@0"
I20260806 23:57:10.899060 14580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:10.921865 14580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:10.924593 14580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:10.924783 14580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:10.925345 14580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:10.926120 14580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:10.926195 14580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.926266 14580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:10.926313 14580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:10.935341 14580 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:34099
I20260806 23:57:10.935364 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:34099 every 8 connection(s)
I20260806 23:57:10.936132 14580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:10.939498 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14580
I20260806 23:57:10.939602 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:10.939867 9009 external_mini_cluster.cc:1428] Setting key 2b1c40a1f157eae21b5614987af54f03
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:10.948347 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.937739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:39543 (local address 127.8.204.126:40279)
0806 23:57:10.937858 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.937862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.938567 (+ 705us) server_negotiation.cc:407] Connection header received
0806 23:57:10.939959 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.939962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.940016 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.940088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.941468 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.942101 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.943116 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.943337 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.943435 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.945493 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.945496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.945499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.945526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.946597 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.947170 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.947172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.947173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.947247 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.947515 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.947517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.947518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.947666 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.947745 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.948147 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.948237 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:57:10.949148 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.938054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40279 (local address 127.8.204.67:39543)
0806 23:57:10.938445 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:10.938482 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:10.939769 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:10.940241 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:10.940262 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:10.940762 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:10.941334 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.941350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.942219 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.942222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.943001 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:10.943010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.943470 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.943474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:10.943626 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.944053 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:10.944067 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:10.945348 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:10.946806 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.946814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.946828 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.947077 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.947323 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:10.947325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:10.947326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:10.947442 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:10.947754 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:10.947758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:10.947950 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 23:57:10.948095 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 23:57:10.948333 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260806 23:57:10.950126 14760 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40279
I20260806 23:57:10.950336 14760 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:10.950858 14760 heartbeater.cc:511] Master 127.8.204.126:40279 requested a full tablet report, sending...
I20260806 23:57:10.951823 14116 ts_manager.cc:194] Registered new tserver with Master: 7c2587a9fe344465937b97e2c01d0fbc (127.8.204.67:34099)
I20260806 23:57:10.952510 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:39543
I20260806 23:57:10.953436 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:57:10.980167 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.971314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.8.204.65:39983)
0806 23:57:10.971630 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.971640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.971662 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:57:10.971713 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.971717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.971872 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.971992 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.972636 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.973339 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.974078 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.974433 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.974511 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.976714 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.976767 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.976778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.976801 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.978406 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.978896 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.978901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.978902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.978976 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.979247 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.979251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.979253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.979469 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.979672 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.979853 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.979996 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
Aug 06 23:57:10 dist-test-slave-39l8 krb5kdc[14044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:10.989966 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:10.982678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.8.204.126:40279)
0806 23:57:10.982832 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:10.982836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:10.982849 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:10.982942 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:10.982944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:10.982982 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:10.983046 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:10.983605 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.984098 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.984875 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:10.985085 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:10.985197 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:10.987466 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:10.987468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:10.987470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:10.987491 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:10.988560 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.988928 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.988930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.988931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.989001 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:10.989215 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:10.989217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:10.989218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:10.989403 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:10.989505 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:10.989680 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 23:57:10.989785 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 23:57:10.990334 14116 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34232
I20260806 23:57:10.990912 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14191
I20260806 23:57:11.002413 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14385
I20260806 23:57:11.012965 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14580
I20260806 23:57:11.023860 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14061
2026-08-06T23:57:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2213 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:11 dist-test-slave-39l8 krb5kdc[14774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:11 dist-test-slave-39l8 krb5kdc[14774](info): set up 2 sockets
Aug 06 23:57:11 dist-test-slave-39l8 krb5kdc[14774](info): commencing operation
krb5kdc: starting...
W20260806 23:57:12.164590 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:12.277052 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35671
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:12.383111 14795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:12.383342 14795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:12.383378 14795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:12.386971 14795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:12.387020 14795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:12.387037 14795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:12.387053 14795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:12.387065 14795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:12.391427 14795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:12.392388 14795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:12.393348 14795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:12.400897 14803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.400897 14806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.400924 14804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:12.401273 14795 server_base.cc:1034] running on GCE node
I20260806 23:57:12.401860 14795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:12.402945 14795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:12.404119 14795 hybrid_clock.cc:648] HybridClock initialized: now 1786060632404101 us; error 29 us; skew 500 ppm
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:12.406591 14795 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:12.407296 14795 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:12.407382 14795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:12.407590 14795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:12.409521 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "16474f5aa12c4d099e91cf0de3458850"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "d318195c476b393a55c1dbf6655907e4"
server_key_iv: "4d4e1afc1b72f895a28f1edcda799d24"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.410054 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "16474f5aa12c4d099e91cf0de3458850"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "d318195c476b393a55c1dbf6655907e4"
server_key_iv: "4d4e1afc1b72f895a28f1edcda799d24"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.413693 14795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:57:12.415995 14817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.417037 14795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:12.417168 14795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "16474f5aa12c4d099e91cf0de3458850"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "d318195c476b393a55c1dbf6655907e4"
server_key_iv: "4d4e1afc1b72f895a28f1edcda799d24"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.417276 14795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:12.426606 14795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:12.429262 14795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:12.429445 14795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:12.435989 14795 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:35671
I20260806 23:57:12.436275 14888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:35671 every 8 connection(s)
I20260806 23:57:12.436650 14795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:12.439249 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:12.442442 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14795
I20260806 23:57:12.442576 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:12.442906 9009 external_mini_cluster.cc:1428] Setting key f93233766d4113107febf1dc4f732dce
I20260806 23:57:12.444880 14889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: Bootstrap starting.
I20260806 23:57:12.447384 14889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:12.448258 14889 log.cc:824] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:12.450240 14889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: No bootstrap required, opened a new log
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:12.454032 14889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } }
I20260806 23:57:12.454252 14889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:12.454313 14889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16474f5aa12c4d099e91cf0de3458850, State: Initialized, Role: FOLLOWER
I20260806 23:57:12.454795 14889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } }
I20260806 23:57:12.454911 14889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:12.454981 14889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:12.455071 14889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:12.455976 14889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } }
I20260806 23:57:12.456336 14889 leader_election.cc:302] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16474f5aa12c4d099e91cf0de3458850; no voters:
I20260806 23:57:12.456676 14889 leader_election.cc:288] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:12.456755 14897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:12.456949 14897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [term 1 LEADER]: Becoming Leader. State: Replica: 16474f5aa12c4d099e91cf0de3458850, State: Running, Role: LEADER
I20260806 23:57:12.456974 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:12.444122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.8.204.126:35671)
0806 23:57:12.444491 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:12.444506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:12.444540 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:57:12.445177 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:12.445195 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:12.445498 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:12.445766 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:12.446613 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.447705 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.448830 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.449336 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.449453 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:12.452186 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:12.452191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:12.452203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:12.452228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:12.453897 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.454297 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.454303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.454310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.454464 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.454762 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.454765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.454767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.455265 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:12.455467 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:12.455779 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 23:57:12.456057 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260806 23:57:12.457293 14897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } }
I20260806 23:57:12.457894 14889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:12.458901 14899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16474f5aa12c4d099e91cf0de3458850. Latest consensus state: current_term: 1 leader_uuid: "16474f5aa12c4d099e91cf0de3458850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } } }
I20260806 23:57:12.459040 14899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:12.459365 14898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16474f5aa12c4d099e91cf0de3458850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16474f5aa12c4d099e91cf0de3458850" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35671 } } }
I20260806 23:57:12.459461 14898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850 [sys.catalog]: This master's current role is: LEADER
W20260806 23:57:12.462911 14919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:57:12.463025 14919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:57:12.463148 14907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:12.463816 14907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:12.469250 14907 catalog_manager.cc:1347] Generated new cluster ID: 25660929d7764330a1c9475bfca35849
I20260806 23:57:12.469317 14907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:12.494218 14907 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:12.495093 14907 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:12.499030 14907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: Generated new TSK 0
I20260806 23:57:12.499619 14907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:12.530776 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:12.651134 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:12.651360 14927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:12.651402 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:12.655201 14927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:12.655246 14927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:12.655360 14927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:12.659965 14927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:12.660825 14927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:12.661757 14927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:12.670305 14927 server_base.cc:1034] running on GCE node
W20260806 23:57:12.670264 14935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.670288 14938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.670272 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:12.670962 14927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:12.671478 14927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:12.672629 14927 hybrid_clock.cc:648] HybridClock initialized: now 1786060632672605 us; error 44 us; skew 500 ppm
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:12.675318 14927 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:12.675954 14927 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:12.676011 14927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:12.676270 14927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:12.678170 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "c10c3412eb1544f694fe97d1cc591654"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "5fdeda7a4406eb5c07c8110d1ee72ed7"
server_key_iv: "1c754c0dfdc7da512c66da3e6e1b3d48"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.678740 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "c10c3412eb1544f694fe97d1cc591654"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "5fdeda7a4406eb5c07c8110d1ee72ed7"
server_key_iv: "1c754c0dfdc7da512c66da3e6e1b3d48"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.682085 14927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:12.684305 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.685396 14927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:12.685524 14927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "c10c3412eb1544f694fe97d1cc591654"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "5fdeda7a4406eb5c07c8110d1ee72ed7"
server_key_iv: "1c754c0dfdc7da512c66da3e6e1b3d48"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.685623 14927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:12.696288 14927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:12.698618 14927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:12.698805 14927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:12.699299 14927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:12.700021 14927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:12.700094 14927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.700166 14927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:12.700203 14927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.708257 14927 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:37685
I20260806 23:57:12.708282 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:37685 every 8 connection(s)
I20260806 23:57:12.708774 14927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:12.717134 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 14927
I20260806 23:57:12.717262 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:12.717522 9009 external_mini_cluster.cc:1428] Setting key 75f4f0506e2cc1762de23b2734cd04fd
I20260806 23:57:12.721338 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:12.710285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35085 (local address 127.8.204.126:35671)
0806 23:57:12.710413 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:12.710418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:12.711603 (+ 1185us) server_negotiation.cc:407] Connection header received
0806 23:57:12.712391 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:12.712394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:12.712439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:12.712508 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:12.713509 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.713985 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.714991 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.715211 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.715284 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:12.717369 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:12.717372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:12.717374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:12.717395 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:12.719293 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.719791 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.719796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.719798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.719908 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.720253 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.720257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.720259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.720449 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:12.720553 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:12.721047 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 23:57:12.721177 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:12.722069 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:12.710632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:35671 (local address 127.8.204.65:35085)
0806 23:57:12.711465 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:12.711504 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:12.712223 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:12.712664 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:12.712670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:12.712983 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:12.713394 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:12.713405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.714080 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.714082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:12.714900 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:12.714910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.715335 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.715337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:12.715470 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:12.715910 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:12.715925 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:12.717196 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:12.719411 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:12.719416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:12.719429 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:12.719686 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:12.720001 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:12.720005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:12.720007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:12.720156 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:12.720554 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:12.720561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:12.720777 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 23:57:12.721035 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 23:57:12.721331 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260806 23:57:12.723188 15115 heartbeater.cc:347] Connected to a master server at 127.8.204.126:35671
I20260806 23:57:12.723397 15115 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:12.723771 15115 heartbeater.cc:511] Master 127.8.204.126:35671 requested a full tablet report, sending...
I20260806 23:57:12.725243 14844 ts_manager.cc:194] Registered new tserver with Master: c10c3412eb1544f694fe97d1cc591654 (127.8.204.65:37685)
I20260806 23:57:12.726307 14844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:35085
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:12.743424 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:12.848874 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:12.849136 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:12.849210 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:12.853461 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:12.853521 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:12.853607 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:12.858350 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:12.859269 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:12.860208 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:12.869005 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.869019 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:12.868994 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:12.869117 15122 server_base.cc:1034] running on GCE node
I20260806 23:57:12.869694 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:12.870237 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:12.871384 15122 hybrid_clock.cc:648] HybridClock initialized: now 1786060632871360 us; error 37 us; skew 500 ppm
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:12.873924 15122 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:12.874562 15122 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:12.874643 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:12.874868 15122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:12.876718 15122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "971ba864ad6047a8888b41d853977484"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "c3ec8cd6de6a3c6a5d60d520bd327d90"
server_key_iv: "6e8288ff3177188ad9a1b88cd0022117"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.877262 15122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "971ba864ad6047a8888b41d853977484"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "c3ec8cd6de6a3c6a5d60d520bd327d90"
server_key_iv: "6e8288ff3177188ad9a1b88cd0022117"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.880610 15122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:12.882861 15143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.883860 15122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:12.883981 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "971ba864ad6047a8888b41d853977484"
format_stamp: "Formatted at 2026-08-06 23:57:12 on dist-test-slave-39l8"
server_key: "c3ec8cd6de6a3c6a5d60d520bd327d90"
server_key_iv: "6e8288ff3177188ad9a1b88cd0022117"
server_key_version: "encryptionkey@0"
I20260806 23:57:12.884075 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:12.896880 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:12.899375 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:12.899550 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:12.900044 15122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:12.900781 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:12.900842 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.900905 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:12.900965 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:12.908936 15122 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:33507
I20260806 23:57:12.908962 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:33507 every 8 connection(s)
I20260806 23:57:12.909444 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
Aug 06 23:57:12 dist-test-slave-39l8 krb5kdc[14774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:12.918927 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 15122
I20260806 23:57:12.919054 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:12.919324 9009 external_mini_cluster.cc:1428] Setting key e9c6a6fcf4401640774aff0a971857ba
I20260806 23:57:12.922390 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:12.911055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49341 (local address 127.8.204.126:35671)
0806 23:57:12.911197 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:12.911201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:12.911961 (+ 760us) server_negotiation.cc:407] Connection header received
0806 23:57:12.913204 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:12.913207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:12.913269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:12.913365 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:12.914467 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.915106 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.916143 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.916350 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.916454 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:12.918738 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:12.918742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:12.918749 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:12.918771 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:12.920290 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.920896 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.920908 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.920910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.921003 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:12.921338 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:12.921342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:12.921344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:12.921518 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:12.921626 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:12.922101 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 23:57:12.922238 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:57:12.923203 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:12.911370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:35671 (local address 127.8.204.66:49341)
0806 23:57:12.911816 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:12.911856 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:12.912995 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:12.913529 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:12.913535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:12.913866 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:12.914320 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:12.914332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.915249 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.915252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:12.916021 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:12.916030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:12.916492 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:12.916496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:12.916648 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:12.917144 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:12.917160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:12.918560 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:12.920402 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:12.920408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:12.920420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:12.920780 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:12.921096 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:12.921100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:12.921102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:12.921241 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:12.921626 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:12.921634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:12.921862 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 23:57:12.922040 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:57:12.922343 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260806 23:57:12.924255 15313 heartbeater.cc:347] Connected to a master server at 127.8.204.126:35671
I20260806 23:57:12.924469 15313 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:12.924855 15313 heartbeater.cc:511] Master 127.8.204.126:35671 requested a full tablet report, sending...
I20260806 23:57:12.925861 14844 ts_manager.cc:194] Registered new tserver with Master: 971ba864ad6047a8888b41d853977484 (127.8.204.66:33507)
I20260806 23:57:12.926676 14844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:49341
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:12.947000 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:13.056051 15320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:13.056272 15320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:13.056311 15320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:13.060091 15320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:13.060135 15320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:13.060189 15320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:13.064985 15320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:35671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:13.065862 15320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:13.066823 15320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:13.075678 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.075699 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.075695 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:13.075856 15320 server_base.cc:1034] running on GCE node
I20260806 23:57:13.076326 15320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:13.076833 15320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:13.078008 15320 hybrid_clock.cc:648] HybridClock initialized: now 1786060633077999 us; error 30 us; skew 500 ppm
Aug 06 23:57:13 dist-test-slave-39l8 krb5kdc[14774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:13.080628 15320 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:13.081327 15320 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:13.081416 15320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:13.081632 15320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:13.083523 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "45b409945e314d0abc73c55e789e36ab"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "940bb47546c3595dfaa05e9e5d300d72"
server_key_iv: "d45ce8a648fd35eda4fbd238b544cbab"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.084040 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "45b409945e314d0abc73c55e789e36ab"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "940bb47546c3595dfaa05e9e5d300d72"
server_key_iv: "d45ce8a648fd35eda4fbd238b544cbab"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.087395 15320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:57:13.089823 15344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.090978 15320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:13.091123 15320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "45b409945e314d0abc73c55e789e36ab"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "940bb47546c3595dfaa05e9e5d300d72"
server_key_iv: "d45ce8a648fd35eda4fbd238b544cbab"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.091229 15320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:13.104507 15320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:13.107173 15320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:13.107360 15320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:13.107882 15320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:13.108680 15320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:13.108749 15320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.108820 15320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:13.108863 15320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.117307 15320 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:45149
I20260806 23:57:13.117326 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:45149 every 8 connection(s)
I20260806 23:57:13.117837 15320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:13.122836 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 15320
I20260806 23:57:13.122961 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:13.123281 9009 external_mini_cluster.cc:1428] Setting key be219e5f6ce97377d08a74b4771a2758
Aug 06 23:57:13 dist-test-slave-39l8 krb5kdc[14774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:13.131207 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.119453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:52855 (local address 127.8.204.126:35671)
0806 23:57:13.119629 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.119634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.120768 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 23:57:13.121632 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.121650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.121723 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.121797 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:13.122906 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.123627 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.125169 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.125404 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.125530 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.127818 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:13.127822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:13.127824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:13.127853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:13.129221 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:13.129735 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:13.129752 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:13.129754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:13.129836 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:13.130155 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:13.130159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:13.130160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:13.130320 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:13.130418 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.130899 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.131037 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 23:57:13.131904 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.119810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:35671 (local address 127.8.204.67:52855)
0806 23:57:13.120631 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.120682 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.121459 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.121947 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.121953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.122293 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:13.122763 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.122784 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.123792 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.123795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.125064 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.125075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.125568 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.125572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.125745 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.126288 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:13.126303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:13.127676 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:13.129364 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:13.129370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:13.129382 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:13.129630 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:13.129921 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:13.129924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:13.129926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:13.130064 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:13.130419 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:13.130426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:13.130640 (+ 214us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.130829 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.131139 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":118,"threads_started":1}
I20260806 23:57:13.132923 15513 heartbeater.cc:347] Connected to a master server at 127.8.204.126:35671
I20260806 23:57:13.133144 15513 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:13.133554 15513 heartbeater.cc:511] Master 127.8.204.126:35671 requested a full tablet report, sending...
I20260806 23:57:13.134454 14844 ts_manager.cc:194] Registered new tserver with Master: 45b409945e314d0abc73c55e789e36ab (127.8.204.67:45149)
I20260806 23:57:13.135033 14844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:52855
I20260806 23:57:13.136777 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:57:13.144884 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.138419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.8.204.126:35671)
0806 23:57:13.138522 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.138526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.138680 (+ 154us) server_negotiation.cc:407] Connection header received
0806 23:57:13.138830 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.138832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.138868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.138939 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:13.139527 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.139964 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.140752 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.140987 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.141090 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.142261 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:13.142265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:13.142267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:13.142290 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:13.143408 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:13.143791 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:13.143794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:13.143795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:13.143868 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:13.144114 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:13.144117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:13.144119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:13.144309 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:13.144435 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.144642 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.144744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:13.147089 14844 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:13.149679 14844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:13.163371 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.157643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33507 (local address 127.0.0.1:47634)
0806 23:57:13.158161 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.158177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.158273 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.159276 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.159278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.159296 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:13.159375 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.159381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.160646 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.160649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.161647 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.161656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162822 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.162827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.162963 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.162988 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.163036 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.163202 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":68,"thread_start_us":83,"threads_started":1}
I20260806 23:57:13.163479 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.157445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.8.204.65:37685)
0806 23:57:13.157889 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.157900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.157923 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:57:13.158006 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.158011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.158204 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.158351 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:13.158970 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.159857 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.161598 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162472 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.162573 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.162705 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.163201 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.163315 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":105,"threads_started":1}
I20260806 23:57:13.163574 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.157236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:37685 (local address 127.0.0.1:59526)
0806 23:57:13.157550 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.157610 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.157724 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.158562 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.158567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.158591 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:13.158801 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.158809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.159971 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.159975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.161461 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.161471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162789 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.162793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.162889 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.162958 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.163049 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.163436 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":284,"thread_start_us":86,"threads_started":1}
I20260806 23:57:13.163368 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.157884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45149 (local address 127.0.0.1:52432)
0806 23:57:13.158471 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.158483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.158562 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.159149 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.159152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.159167 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:13.159258 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.159264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.160577 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.160580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.161812 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.161822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162979 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.162981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.163067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.163085 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.163142 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.163208 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":46,"thread_start_us":110,"threads_started":1}
I20260806 23:57:13.164021 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.158051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.8.204.66:33507)
0806 23:57:13.158836 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.158858 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.158878 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:57:13.158998 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.159003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.159163 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.159294 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:13.159475 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.160492 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.161787 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162628 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.163156 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.163305 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.163356 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.163443 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":157,"threads_started":1}
I20260806 23:57:13.164201 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.158283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.8.204.67:45149)
0806 23:57:13.158577 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.158599 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.158622 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:57:13.158690 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.158695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.158885 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.159015 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:13.159368 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.160447 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.161942 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.162674 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.162802 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.162922 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.163977 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.164063 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260806 23:57:13.166234 15212 tablet_service.cc:1467] Processing CreateTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 (DEFAULT_TABLE table=test-table [id=928850f82c0142ea850886a0d8b64321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:13.166389 15020 tablet_service.cc:1467] Processing CreateTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 (DEFAULT_TABLE table=test-table [id=928850f82c0142ea850886a0d8b64321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:13.166519 15418 tablet_service.cc:1467] Processing CreateTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 (DEFAULT_TABLE table=test-table [id=928850f82c0142ea850886a0d8b64321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:13.167268 15212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c96bbcf9e244d9abe230145aae3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:13.167414 15418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c96bbcf9e244d9abe230145aae3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:13.167411 15020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c96bbcf9e244d9abe230145aae3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:13.172899 15536 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Bootstrap starting.
I20260806 23:57:13.174161 15537 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Bootstrap starting.
I20260806 23:57:13.174777 15538 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Bootstrap starting.
I20260806 23:57:13.175618 15536 tablet_bootstrap.cc:654] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:13.176319 15537 tablet_bootstrap.cc:654] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:13.176625 15538 tablet_bootstrap.cc:654] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:13.177006 15536 log.cc:824] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:13.177428 15537 log.cc:824] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:13.177680 15538 log.cc:824] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:13.178867 15536 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: No bootstrap required, opened a new log
I20260806 23:57:13.179065 15536 ts_tablet_manager.cc:1397] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:57:13.179394 15538 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: No bootstrap required, opened a new log
I20260806 23:57:13.179586 15538 ts_tablet_manager.cc:1397] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:13.180913 15537 tablet_bootstrap.cc:492] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: No bootstrap required, opened a new log
I20260806 23:57:13.181123 15537 ts_tablet_manager.cc:1397] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:57:13.182137 15536 raft_consensus.cc:348] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.182364 15536 raft_consensus.cc:374] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:13.182430 15536 raft_consensus.cc:729] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ba864ad6047a8888b41d853977484, State: Initialized, Role: FOLLOWER
I20260806 23:57:13.182878 15536 consensus_queue.cc:260] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.183286 15538 raft_consensus.cc:348] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.183491 15313 heartbeater.cc:503] Master 127.8.204.126:35671 was elected leader, sending a full tablet report...
I20260806 23:57:13.183615 15538 raft_consensus.cc:374] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:13.183723 15538 raft_consensus.cc:729] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b409945e314d0abc73c55e789e36ab, State: Initialized, Role: FOLLOWER
I20260806 23:57:13.183844 15536 ts_tablet_manager.cc:1428] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:13.184262 15538 consensus_queue.cc:260] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.185014 15513 heartbeater.cc:503] Master 127.8.204.126:35671 was elected leader, sending a full tablet report...
I20260806 23:57:13.185165 15538 ts_tablet_manager.cc:1428] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:57:13.185237 15537 raft_consensus.cc:348] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.185449 15537 raft_consensus.cc:374] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:13.185487 15537 raft_consensus.cc:729] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10c3412eb1544f694fe97d1cc591654, State: Initialized, Role: FOLLOWER
I20260806 23:57:13.185853 15537 consensus_queue.cc:260] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.186692 15115 heartbeater.cc:503] Master 127.8.204.126:35671 was elected leader, sending a full tablet report...
I20260806 23:57:13.187726 15537 ts_tablet_manager.cc:1428] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:57:13.188510 15544 raft_consensus.cc:482] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:13.188665 15544 raft_consensus.cc:504] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.189708 15544 leader_election.cc:288] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c10c3412eb1544f694fe97d1cc591654 (127.8.204.65:37685), 971ba864ad6047a8888b41d853977484 (127.8.204.66:33507)
I20260806 23:57:13.193945 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.190302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:48623 (local address 127.8.204.66:33507)
0806 23:57:13.190401 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.190406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.190417 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:57:13.190458 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.190461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.190500 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.190574 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:13.190872 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.191459 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.192738 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.193326 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.193472 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.193521 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.193769 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.193835 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 23:57:13.193945 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.190425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:58091 (local address 127.8.204.65:37685)
0806 23:57:13.190515 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.190518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.190526 (+ 8us) server_negotiation.cc:407] Connection header received
0806 23:57:13.190577 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.190579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.190617 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.190679 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:13.191077 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.191564 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.192778 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.193389 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.193490 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.193532 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.193769 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.193835 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 23:57:13.193996 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.189938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33507 (local address 127.8.204.67:48623)
0806 23:57:13.190235 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.190248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.190330 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.190591 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.190594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.190608 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:13.190774 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.190782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.191561 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.191564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.192604 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.192616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.193467 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.193470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.193609 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.193664 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.193742 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.193830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":99,"thread_start_us":91,"threads_started":1}
I20260806 23:57:13.193996 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.189850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:37685 (local address 127.8.204.67:58091)
0806 23:57:13.190147 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.190163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.190244 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.190800 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.190807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.190833 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:13.190953 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.190959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.191677 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.191680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.192618 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:13.192629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.193483 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.193485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:13.193615 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.193659 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.193742 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.193830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":67}
I20260806 23:57:13.195012 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c96bbcf9e244d9abe230145aae3c54" candidate_uuid: "45b409945e314d0abc73c55e789e36ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10c3412eb1544f694fe97d1cc591654" is_pre_election: true
I20260806 23:57:13.195019 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c96bbcf9e244d9abe230145aae3c54" candidate_uuid: "45b409945e314d0abc73c55e789e36ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971ba864ad6047a8888b41d853977484" is_pre_election: true
I20260806 23:57:13.195276 15049 raft_consensus.cc:2454] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b409945e314d0abc73c55e789e36ab in term 0.
I20260806 23:57:13.195281 15245 raft_consensus.cc:2454] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b409945e314d0abc73c55e789e36ab in term 0.
I20260806 23:57:13.195758 15352 leader_election.cc:302] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b409945e314d0abc73c55e789e36ab, 971ba864ad6047a8888b41d853977484; no voters:
I20260806 23:57:13.195964 15544 raft_consensus.cc:2781] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:13.196033 15544 raft_consensus.cc:482] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:13.196072 15544 raft_consensus.cc:3037] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:13.196975 15544 raft_consensus.cc:504] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.197263 15544 leader_election.cc:288] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [CANDIDATE]: Term 1 election: Requested vote from peers c10c3412eb1544f694fe97d1cc591654 (127.8.204.65:37685), 971ba864ad6047a8888b41d853977484 (127.8.204.66:33507)
I20260806 23:57:13.197558 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c96bbcf9e244d9abe230145aae3c54" candidate_uuid: "45b409945e314d0abc73c55e789e36ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10c3412eb1544f694fe97d1cc591654"
I20260806 23:57:13.197588 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c96bbcf9e244d9abe230145aae3c54" candidate_uuid: "45b409945e314d0abc73c55e789e36ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971ba864ad6047a8888b41d853977484"
I20260806 23:57:13.197680 15049 raft_consensus.cc:3037] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:13.197705 15245 raft_consensus.cc:3037] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:13.198628 15049 raft_consensus.cc:2454] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b409945e314d0abc73c55e789e36ab in term 1.
I20260806 23:57:13.198815 15245 raft_consensus.cc:2454] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b409945e314d0abc73c55e789e36ab in term 1.
I20260806 23:57:13.198959 15347 leader_election.cc:302] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b409945e314d0abc73c55e789e36ab, c10c3412eb1544f694fe97d1cc591654; no voters:
I20260806 23:57:13.199170 15544 raft_consensus.cc:2781] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:13.199368 15544 raft_consensus.cc:686] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 1 LEADER]: Becoming Leader. State: Replica: 45b409945e314d0abc73c55e789e36ab, State: Running, Role: LEADER
I20260806 23:57:13.199658 15544 consensus_queue.cc:237] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } }
I20260806 23:57:13.202899 14843 catalog_manager.cc:5495] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b409945e314d0abc73c55e789e36ab (127.8.204.67). New cstate: current_term: 1 leader_uuid: "45b409945e314d0abc73c55e789e36ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45b409945e314d0abc73c55e789e36ab" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 } health_report { overall_health: UNKNOWN } } }
W20260806 23:57:13.209672 15116 tablet.cc:2369] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:13.218904 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.215418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.8.204.67:45149)
0806 23:57:13.215545 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.215550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.215609 (+ 59us) server_negotiation.cc:407] Connection header received
0806 23:57:13.215784 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.215787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.215841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.215941 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:13.216191 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.216674 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.217507 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:13.217730 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:13.217832 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:13.218112 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:13.218485 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:13.218613 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.218681 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.218758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:57:13.224092 15049 raft_consensus.cc:1261] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 1 FOLLOWER]: Refusing update from remote peer 45b409945e314d0abc73c55e789e36ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:13.224094 15245 raft_consensus.cc:1261] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 1 FOLLOWER]: Refusing update from remote peer 45b409945e314d0abc73c55e789e36ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:13.224782 15547 consensus_queue.cc:1035] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "971ba864ad6047a8888b41d853977484" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:13.224946 15544 consensus_queue.cc:1035] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c10c3412eb1544f694fe97d1cc591654" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:13.232337 15552 mvcc.cc:205] Tried to move back new op lower bound from 7315704353679720448 to 7315704353588572160. Current Snapshot: MvccSnapshot[applied={T|T < 7315704353679720448}]
I20260806 23:57:13.234678 15554 mvcc.cc:205] Tried to move back new op lower bound from 7315704353679720448 to 7315704353588572160. Current Snapshot: MvccSnapshot[applied={T|T < 7315704353679720448}]
I20260806 23:57:13.240123 14843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:13.240252 14843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:13.242564 14843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16474f5aa12c4d099e91cf0de3458850: Sending DeleteTablet for 3 replicas of tablet 99c96bbcf9e244d9abe230145aae3c54
I20260806 23:57:13.243350 15212 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:13 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:47634
I20260806 23:57:13.243604 15418 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:13 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:52432
I20260806 23:57:13.243789 15566 tablet_replica.cc:330] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: stopping tablet replica
I20260806 23:57:13.243803 15020 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c96bbcf9e244d9abe230145aae3c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:13 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:59526
I20260806 23:57:13.244089 15566 raft_consensus.cc:2229] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:13.244220 15569 tablet_replica.cc:330] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654: stopping tablet replica
I20260806 23:57:13.244417 15566 raft_consensus.cc:2258] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:13.244433 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14927
I20260806 23:57:13.244562 15569 raft_consensus.cc:2229] T 99c96bbcf9e244d9abe230145aae3c54 P c10c3412eb1544f694fe97d1cc591654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:13.264583 15567 tablet_replica.cc:330] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: stopping tablet replica
I20260806 23:57:13.265040 15567 raft_consensus.cc:2229] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:13.265468 15567 raft_consensus.cc:2258] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:13.265545 15566 ts_tablet_manager.cc:1905] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:13.267706 15567 ts_tablet_manager.cc:1905] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:13.267805 15566 ts_tablet_manager.cc:1918] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:13.267922 15566 log.cc:1196] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/99c96bbcf9e244d9abe230145aae3c54
I20260806 23:57:13.268322 15566 ts_tablet_manager.cc:1939] T 99c96bbcf9e244d9abe230145aae3c54 P 971ba864ad6047a8888b41d853977484: Deleting consensus metadata
I20260806 23:57:13.269622 14822 catalog_manager.cc:4841] TS 971ba864ad6047a8888b41d853977484 (127.8.204.66:33507): tablet 99c96bbcf9e244d9abe230145aae3c54 (table test-table [id=928850f82c0142ea850886a0d8b64321]) successfully deleted
I20260806 23:57:13.271263 15567 ts_tablet_manager.cc:1918] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:13.271394 15567 log.cc:1196] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/99c96bbcf9e244d9abe230145aae3c54
I20260806 23:57:13.271760 15567 ts_tablet_manager.cc:1939] T 99c96bbcf9e244d9abe230145aae3c54 P 45b409945e314d0abc73c55e789e36ab: Deleting consensus metadata
W20260806 23:57:13.272097 14820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:37685 (error 108)
I20260806 23:57:13.273065 14820 catalog_manager.cc:4841] TS 45b409945e314d0abc73c55e789e36ab (127.8.204.67:45149): tablet 99c96bbcf9e244d9abe230145aae3c54 (table test-table [id=928850f82c0142ea850886a0d8b64321]) successfully deleted
I20260806 23:57:13.273208 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.272919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:37685 (local address 127.0.0.1:59532)
0806 23:57:13.273039 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.273123 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:37685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 23:57:13.273327 14820 catalog_manager.cc:4584] TS c10c3412eb1544f694fe97d1cc591654 (127.8.204.65:37685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99c96bbcf9e244d9abe230145aae3c54: Network error: Client connection negotiation failed: client connection to 127.8.204.65:37685: connect: Connection refused (error 111)
I20260806 23:57:13.273764 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 15122
I20260806 23:57:13.288146 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 15320
I20260806 23:57:13.298995 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 14795
2026-08-06T23:57:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2275 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T23:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:13Z Disabled control of system clock
I20260806 23:57:13.332935 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41531
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 23:57:13.437151 15577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:13.437388 15577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:13.437422 15577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:13.441067 15577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:13.441116 15577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:13.441131 15577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:13.441147 15577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:13.441162 15577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:13.445411 15577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:41531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:13.446203 15577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:13.447415 15577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:13.455534 15577 server_base.cc:1034] running on GCE node
W20260806 23:57:13.455475 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.455467 15585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.455446 15584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:13.456182 15577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:13.457389 15577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:13.458565 15577 hybrid_clock.cc:648] HybridClock initialized: now 1786060633458536 us; error 41 us; skew 500 ppm
I20260806 23:57:13.460240 15577 webserver.cc:460] Webserver started at http://127.8.204.126:41491/ using document root <none> and password file <none>
I20260806 23:57:13.460747 15577 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:13.460803 15577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:13.461037 15577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:13.463024 15577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "074b1d25fdf14738a35d76420248eb5d"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "dc71ac7f772da7a94ec54e3007d518c5"
server_key_iv: "7b6ee90c37584979bb786a7ef1acd02a"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.463582 15577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "074b1d25fdf14738a35d76420248eb5d"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "dc71ac7f772da7a94ec54e3007d518c5"
server_key_iv: "7b6ee90c37584979bb786a7ef1acd02a"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.467083 15577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:13.469282 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.470357 15577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:13.470501 15577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "074b1d25fdf14738a35d76420248eb5d"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "dc71ac7f772da7a94ec54e3007d518c5"
server_key_iv: "7b6ee90c37584979bb786a7ef1acd02a"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.470597 15577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:13.474907 15577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:13.475451 15577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:13.475622 15577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:13.482179 15577 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:41531
I20260806 23:57:13.482200 15668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:41531 every 8 connection(s)
I20260806 23:57:13.483078 15577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:13.485998 15669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:13.487392 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 15577
I20260806 23:57:13.487526 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:13.487775 9009 external_mini_cluster.cc:1428] Setting key f65b86555d078d8364ef641a2dff32ef
I20260806 23:57:13.492036 15669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: Bootstrap starting.
I20260806 23:57:13.492875 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.489051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.8.204.126:41531)
0806 23:57:13.489600 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.489613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.489646 (+ 33us) server_negotiation.cc:407] Connection header received
0806 23:57:13.490255 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.490271 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.490565 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.490868 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:13.491678 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:13.491683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:13.491694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:13.491704 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:13.491753 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:13.491768 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:13.491871 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.492006 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.492260 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260806 23:57:13.494325 15669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:13.495146 15669 log.cc:824] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:13.496858 15669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: No bootstrap required, opened a new log
I20260806 23:57:13.500275 15669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } }
I20260806 23:57:13.500490 15669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:13.500562 15669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074b1d25fdf14738a35d76420248eb5d, State: Initialized, Role: FOLLOWER
I20260806 23:57:13.500984 15669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } }
I20260806 23:57:13.501153 15669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:13.501240 15669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:13.501338 15669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:13.502126 15669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } }
I20260806 23:57:13.502488 15669 leader_election.cc:302] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074b1d25fdf14738a35d76420248eb5d; no voters:
I20260806 23:57:13.502816 15669 leader_election.cc:288] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:13.502883 15675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:13.503159 15675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [term 1 LEADER]: Becoming Leader. State: Replica: 074b1d25fdf14738a35d76420248eb5d, State: Running, Role: LEADER
I20260806 23:57:13.503463 15675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } }
I20260806 23:57:13.504089 15669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:13.505002 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074b1d25fdf14738a35d76420248eb5d. Latest consensus state: current_term: 1 leader_uuid: "074b1d25fdf14738a35d76420248eb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } } }
I20260806 23:57:13.505162 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:13.505448 15676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074b1d25fdf14738a35d76420248eb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074b1d25fdf14738a35d76420248eb5d" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41531 } } }
I20260806 23:57:13.505520 15676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:13.505584 15689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:13.508569 15689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:13.513571 15689 catalog_manager.cc:1347] Generated new cluster ID: 1d264fa6b6c7467d8bb31a65ac9a4332
I20260806 23:57:13.513638 15689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:13.524657 15689 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:13.524938 15689 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:13.531869 15689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: Generated new TSK 0
I20260806 23:57:13.532555 15689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:13.544165 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--builtin_ntp_servers=127.8.204.84:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 23:57:13.656807 15701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:13.657020 15701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:13.657054 15701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:13.660602 15701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:13.660645 15701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:13.660693 15701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:13.665143 15701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:13.665978 15701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:13.667187 15701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:13.676515 15712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.676515 15708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.676523 15707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:13.676975 15701 server_base.cc:1034] running on GCE node
I20260806 23:57:13.677366 15701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:13.677934 15701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:13.679116 15701 hybrid_clock.cc:648] HybridClock initialized: now 1786060633679104 us; error 24 us; skew 500 ppm
I20260806 23:57:13.680867 15701 webserver.cc:460] Webserver started at http://127.8.204.65:39069/ using document root <none> and password file <none>
I20260806 23:57:13.681458 15701 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:13.681553 15701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:13.681777 15701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:13.683780 15701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "a6ace18930734c1e8f0c6ab11c5871ba"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "0a4b7a61c23ba7bcd080ddeb86c831af"
server_key_iv: "098bb3be43e40d4ffa1e4e282cfaeb54"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.684337 15701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "a6ace18930734c1e8f0c6ab11c5871ba"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "0a4b7a61c23ba7bcd080ddeb86c831af"
server_key_iv: "098bb3be43e40d4ffa1e4e282cfaeb54"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.687713 15701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:13.689903 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.691119 15701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:13.691248 15701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "a6ace18930734c1e8f0c6ab11c5871ba"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "0a4b7a61c23ba7bcd080ddeb86c831af"
server_key_iv: "098bb3be43e40d4ffa1e4e282cfaeb54"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.691352 15701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:13.695434 15701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:13.695961 15701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:13.696133 15701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:13.696621 15701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:13.697422 15701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:13.697484 15701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.697542 15701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:13.697587 15701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.705857 15701 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:44827
I20260806 23:57:13.705904 15880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:44827 every 8 connection(s)
I20260806 23:57:13.706619 15701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:13.711228 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 15701
I20260806 23:57:13.711362 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:13.711690 9009 external_mini_cluster.cc:1428] Setting key 2061504be8118d96faaaf7c1ace21b85
I20260806 23:57:13.712553 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.708220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:47759 (local address 127.8.204.126:41531)
0806 23:57:13.708357 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.708362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.709317 (+ 955us) server_negotiation.cc:407] Connection header received
0806 23:57:13.710668 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.710671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.710755 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.710832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:13.712037 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:13.712043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:13.712049 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:13.712066 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:13.712090 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:13.712105 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:13.712189 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.712370 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.712414 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:13.713431 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.708587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41531 (local address 127.8.204.65:47759)
0806 23:57:13.709185 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.709227 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.710375 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.710991 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.710997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.711097 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:13.711716 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:13.711718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:13.711782 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:13.711854 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:13.712200 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:13.712210 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:13.712217 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.712382 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.712591 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
I20260806 23:57:13.713794 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--builtin_ntp_servers=127.8.204.84:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 23:57:13.714511 15882 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41531
I20260806 23:57:13.714696 15882 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:13.715067 15882 heartbeater.cc:511] Master 127.8.204.126:41531 requested a full tablet report, sending...
I20260806 23:57:13.716368 15623 ts_manager.cc:194] Registered new tserver with Master: a6ace18930734c1e8f0c6ab11c5871ba (127.8.204.65:44827)
W20260806 23:57:13.819367 15885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:13.819587 15885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:13.819659 15885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:13.823338 15885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:13.823374 15885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:13.823419 15885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:13.827648 15885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:13.828473 15885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:13.829610 15885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:13.838500 15885 server_base.cc:1034] running on GCE node
W20260806 23:57:13.838411 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.838429 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:13.838433 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:13.839056 15885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:13.839618 15885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:13.840770 15885 hybrid_clock.cc:648] HybridClock initialized: now 1786060633840738 us; error 41 us; skew 500 ppm
I20260806 23:57:13.842502 15885 webserver.cc:460] Webserver started at http://127.8.204.66:42961/ using document root <none> and password file <none>
I20260806 23:57:13.843087 15885 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:13.843146 15885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:13.843364 15885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:13.845155 15885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "d62a9f6042d548d585f45e196430a1ef"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "b1fc339c83acec13a69eeee12bb05377"
server_key_iv: "1ae0fe0b7cfef70deaa95b58d8af365b"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.845657 15885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "d62a9f6042d548d585f45e196430a1ef"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "b1fc339c83acec13a69eeee12bb05377"
server_key_iv: "1ae0fe0b7cfef70deaa95b58d8af365b"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.848858 15885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 23:57:13.851215 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.852339 15885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:13.852465 15885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "d62a9f6042d548d585f45e196430a1ef"
format_stamp: "Formatted at 2026-08-06 23:57:13 on dist-test-slave-39l8"
server_key: "b1fc339c83acec13a69eeee12bb05377"
server_key_iv: "1ae0fe0b7cfef70deaa95b58d8af365b"
server_key_version: "encryptionkey@0"
I20260806 23:57:13.852566 15885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:13.856609 15885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:13.857102 15885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:13.857266 15885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:13.857745 15885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:13.858487 15885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:13.858559 15885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.858642 15885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:13.858683 15885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:13.866984 15885 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:33669
I20260806 23:57:13.867010 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:33669 every 8 connection(s)
I20260806 23:57:13.867740 15885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:13.870462 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 15885
I20260806 23:57:13.870582 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:13.870865 9009 external_mini_cluster.cc:1428] Setting key 9bd619b6a986c6398cb4c4cb019a795d
I20260806 23:57:13.873055 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--builtin_ntp_servers=127.8.204.84:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 23:57:13.873906 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.869365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:60237 (local address 127.8.204.126:41531)
0806 23:57:13.869519 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:13.869524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:13.870483 (+ 959us) server_negotiation.cc:407] Connection header received
0806 23:57:13.872047 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:13.872050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:13.872103 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:13.872194 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:13.873414 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:13.873420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:13.873425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:13.873444 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:13.873470 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:13.873484 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:13.873576 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:13.873675 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 23:57:13.873731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:57:13.874814 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:13.869759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41531 (local address 127.8.204.66:60237)
0806 23:57:13.870356 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:13.870389 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:13.871829 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:13.872387 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:13.872394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:13.872496 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:13.873072 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:13.873073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:13.873109 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:13.873195 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:13.873565 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:13.873572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:13.873577 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:57:13.873711 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 23:57:13.873889 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260806 23:57:13.875857 16063 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41531
I20260806 23:57:13.876087 16063 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:13.876561 16063 heartbeater.cc:511] Master 127.8.204.126:41531 requested a full tablet report, sending...
I20260806 23:57:13.877595 15623 ts_manager.cc:194] Registered new tserver with Master: d62a9f6042d548d585f45e196430a1ef (127.8.204.66:33669)
W20260806 23:57:13.982800 16066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:13.983052 16066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:13.983134 16066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:13.986879 16066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:13.986922 16066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:13.986997 16066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:13.991681 16066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:13.992672 16066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:13.993942 16066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:14.002818 16076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.002811 16075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.002811 16078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:14.003055 16066 server_base.cc:1034] running on GCE node
I20260806 23:57:14.003409 16066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:14.004014 16066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:14.005220 16066 hybrid_clock.cc:648] HybridClock initialized: now 1786060634005208 us; error 24 us; skew 500 ppm
I20260806 23:57:14.006942 16066 webserver.cc:460] Webserver started at http://127.8.204.67:35473/ using document root <none> and password file <none>
I20260806 23:57:14.007486 16066 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:14.007572 16066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:14.007790 16066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:14.009815 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "4241c3db55754126ab837fab1c79b63f"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "489d7d6d482cb1e43eb7f457f19d327e"
server_key_iv: "fbf70a8673200f69746bc0eef9185fb9"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.010368 16066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "4241c3db55754126ab837fab1c79b63f"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "489d7d6d482cb1e43eb7f457f19d327e"
server_key_iv: "fbf70a8673200f69746bc0eef9185fb9"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.014971 16066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 23:57:14.018754 16088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:14.019862 16066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:57:14.019974 16066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "4241c3db55754126ab837fab1c79b63f"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "489d7d6d482cb1e43eb7f457f19d327e"
server_key_iv: "fbf70a8673200f69746bc0eef9185fb9"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.020076 16066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:14.024513 16066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:14.025072 16066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:14.025249 16066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:14.025789 16066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:14.026652 16066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:14.026697 16066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:14.026813 16066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:14.026847 16066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:14.035578 16066 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:40919
I20260806 23:57:14.035637 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:40919 every 8 connection(s)
I20260806 23:57:14.036396 16066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:14.040808 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16066
I20260806 23:57:14.040895 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:14.041260 9009 external_mini_cluster.cc:1428] Setting key 62b7574762069bce149dde7ddbb71854
I20260806 23:57:14.042254 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.038040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:43025 (local address 127.8.204.126:41531)
0806 23:57:14.038198 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.038203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.039238 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 23:57:14.040423 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.040426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.040463 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.040521 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.041691 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.041694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.041696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.041723 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.041749 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.041759 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.041845 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.042030 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.042088 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:57:14.043193 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.038463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41531 (local address 127.8.204.67:43025)
0806 23:57:14.039108 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.039145 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.040230 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.040662 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.040668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.040741 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.041415 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.041417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.041452 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.041518 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.041857 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.041864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.041886 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.042076 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.042350 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260806 23:57:14.044267 16251 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41531
I20260806 23:57:14.044534 16251 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:14.044916 16251 heartbeater.cc:511] Master 127.8.204.126:41531 requested a full tablet report, sending...
I20260806 23:57:14.045874 15623 ts_manager.cc:194] Registered new tserver with Master: 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919)
I20260806 23:57:14.055078 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:57:14.058312 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.056940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.8.204.126:41531)
0806 23:57:14.057047 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.057051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.057269 (+ 218us) server_negotiation.cc:407] Connection header received
0806 23:57:14.057350 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.057352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.057382 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.057446 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.058070 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.058072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.058074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.058081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.058094 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.058100 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.058155 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.058199 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.058228 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:57:14.060811 15623 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:14.063654 15623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:14.075124 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.071606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33669 (local address 127.0.0.1:56594)
0806 23:57:14.072267 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.072332 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.072430 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.073269 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.073278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.073356 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.074005 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.074009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.074032 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.074131 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.074760 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.074771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.074777 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.074872 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.074956 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260806 23:57:14.075181 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.071720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:40919 (local address 127.0.0.1:54274)
0806 23:57:14.072787 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.072800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.072888 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.073347 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.073350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.073360 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.074053 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.074055 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.074147 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.074169 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.074766 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.074772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.074777 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.074872 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.075000 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
I20260806 23:57:14.075299 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.072430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.8.204.67:40919)
0806 23:57:14.072736 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.072746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.072890 (+ 144us) server_negotiation.cc:407] Connection header received
0806 23:57:14.073023 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.073035 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.073213 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.073640 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.074527 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.074533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.074544 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.074554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.074595 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.074622 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.075002 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.075050 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.075133 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260806 23:57:14.075332 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.072494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44827 (local address 127.0.0.1:33052)
0806 23:57:14.073080 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.073092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.073160 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.074106 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.074126 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.074141 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.074579 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.074583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.074596 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.074629 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.075108 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.075111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.075115 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.075168 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.075213 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":89,"threads_started":1}
I20260806 23:57:14.075923 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.073095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.8.204.65:44827)
0806 23:57:14.073675 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.073686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.073708 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:57:14.073763 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.073769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.073933 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.074136 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.074917 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.074921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.074927 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.074934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.074982 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.075005 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.075581 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.075630 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.075713 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":114,"threads_started":1}
I20260806 23:57:14.076418 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.071746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.8.204.66:33669)
0806 23:57:14.072526 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.072548 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.072598 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:57:14.072655 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.072660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.072862 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.073640 (+ 778us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.074528 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.074533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.074544 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.074554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.074601 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.074622 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.076148 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.076198 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.076262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":124,"threads_started":1}
I20260806 23:57:14.078125 16155 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 (DEFAULT_TABLE table=test-table [id=c4f009342ec34adba55e206b636e6e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:14.078549 15786 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 (DEFAULT_TABLE table=test-table [id=c4f009342ec34adba55e206b636e6e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:14.079206 16155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5ef4adcd644f4a90d39672e723fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:14.078893 15969 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 (DEFAULT_TABLE table=test-table [id=c4f009342ec34adba55e206b636e6e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:14.079579 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5ef4adcd644f4a90d39672e723fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:14.079752 15969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5ef4adcd644f4a90d39672e723fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:14.085294 16274 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Bootstrap starting.
I20260806 23:57:14.086753 16275 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Bootstrap starting.
I20260806 23:57:14.086759 16277 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Bootstrap starting.
I20260806 23:57:14.087731 16274 tablet_bootstrap.cc:654] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:14.088631 16274 log.cc:824] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:14.088778 16277 tablet_bootstrap.cc:654] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:14.088778 16275 tablet_bootstrap.cc:654] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:14.089592 16275 log.cc:824] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:14.089592 16277 log.cc:824] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:14.090556 16274 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: No bootstrap required, opened a new log
I20260806 23:57:14.090793 16274 ts_tablet_manager.cc:1397] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:57:14.091476 16277 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: No bootstrap required, opened a new log
I20260806 23:57:14.091476 16275 tablet_bootstrap.cc:492] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: No bootstrap required, opened a new log
I20260806 23:57:14.091660 16275 ts_tablet_manager.cc:1397] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:14.091660 16277 ts_tablet_manager.cc:1397] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:14.094492 16274 raft_consensus.cc:348] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.094744 16274 raft_consensus.cc:374] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:14.094799 16274 raft_consensus.cc:729] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4241c3db55754126ab837fab1c79b63f, State: Initialized, Role: FOLLOWER
I20260806 23:57:14.095271 16274 consensus_queue.cc:260] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.095964 16251 heartbeater.cc:503] Master 127.8.204.126:41531 was elected leader, sending a full tablet report...
I20260806 23:57:14.095997 16277 raft_consensus.cc:348] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.096231 16277 raft_consensus.cc:374] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:14.096287 16277 raft_consensus.cc:729] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62a9f6042d548d585f45e196430a1ef, State: Initialized, Role: FOLLOWER
I20260806 23:57:14.096321 16274 ts_tablet_manager.cc:1428] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:57:14.096647 16277 consensus_queue.cc:260] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.096709 16275 raft_consensus.cc:348] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.096952 16275 raft_consensus.cc:374] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:14.097013 16275 raft_consensus.cc:729] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ace18930734c1e8f0c6ab11c5871ba, State: Initialized, Role: FOLLOWER
I20260806 23:57:14.097348 16063 heartbeater.cc:503] Master 127.8.204.126:41531 was elected leader, sending a full tablet report...
I20260806 23:57:14.097393 16275 consensus_queue.cc:260] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.098030 15882 heartbeater.cc:503] Master 127.8.204.126:41531 was elected leader, sending a full tablet report...
I20260806 23:57:14.098301 16275 ts_tablet_manager.cc:1428] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:57:14.098860 16277 ts_tablet_manager.cc:1428] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 23:57:14.118827 16064 tablet.cc:2369] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:14.170320 16282 raft_consensus.cc:482] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:14.170578 16282 raft_consensus.cc:504] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.171636 16284 raft_consensus.cc:482] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:14.171775 16282 leader_election.cc:288] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919), a6ace18930734c1e8f0c6ab11c5871ba (127.8.204.65:44827)
I20260806 23:57:14.171856 16284 raft_consensus.cc:504] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.173629 16284 leader_election.cc:288] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62a9f6042d548d585f45e196430a1ef (127.8.204.66:33669), 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919)
I20260806 23:57:14.174041 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.172417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49849 (local address 127.8.204.65:44827)
0806 23:57:14.172526 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.172531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.172619 (+ 88us) server_negotiation.cc:407] Connection header received
0806 23:57:14.172776 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.172780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.172820 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.172879 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.173612 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.173615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.173617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.173626 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.173645 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.173656 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.173710 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.173875 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.173917 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:14.174398 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.172015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:40919 (local address 127.8.204.66:58507)
0806 23:57:14.172218 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.172234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.172318 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.172747 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.172753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.172766 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.173445 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.173447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.173459 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.173487 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.174208 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.174213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.174219 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.174289 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.174310 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 23:57:14.174427 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.172062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44827 (local address 127.8.204.66:49849)
0806 23:57:14.172518 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.172532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.172651 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.173000 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.173002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.173013 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.173477 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.173478 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.173485 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.173500 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.173777 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.173780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.173783 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.174245 (+ 462us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.174283 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":138,"threads_started":1}
I20260806 23:57:14.174642 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.172156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:58507 (local address 127.8.204.67:40919)
0806 23:57:14.172493 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.172498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.172513 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:14.172559 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.172562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.172612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.172693 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.173973 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.173978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.173984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.173998 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.174015 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.174025 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.174102 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.174472 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.174515 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260806 23:57:14.175233 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.173299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33669 (local address 127.8.204.65:54353)
0806 23:57:14.173634 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.173646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.173712 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.173980 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.173983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.173994 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.174591 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.174595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.174609 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.174640 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.175014 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.175017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.175020 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.175091 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.175123 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260806 23:57:14.175449 15819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "d62a9f6042d548d585f45e196430a1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" is_pre_election: true
I20260806 23:57:14.175523 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.173342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:54353 (local address 127.8.204.66:33669)
0806 23:57:14.173458 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.173464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.173783 (+ 319us) server_negotiation.cc:407] Connection header received
0806 23:57:14.173812 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.173817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.173850 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.173910 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.174845 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.174850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.174854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.174869 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.174891 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.174903 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.174978 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.175378 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.175414 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:14.175660 15819 raft_consensus.cc:2454] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62a9f6042d548d585f45e196430a1ef in term 0.
I20260806 23:57:14.175705 16186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "d62a9f6042d548d585f45e196430a1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241c3db55754126ab837fab1c79b63f" is_pre_election: true
I20260806 23:57:14.175935 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.173664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:56661 (local address 127.8.204.67:40919)
0806 23:57:14.174334 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.174344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.174356 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:14.174385 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.174388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.174419 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.174885 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.175307 (+ 422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.175310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.175312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.175323 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.175346 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.175354 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.175729 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.175765 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.175800 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":140,"threads_started":1}
I20260806 23:57:14.175987 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.173370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:40919 (local address 127.8.204.65:56661)
0806 23:57:14.173930 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.173944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.174021 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.174568 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.174571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.174596 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:14.175103 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:14.175105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:14.175115 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:14.175148 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:14.175503 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:14.175506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:14.175510 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:57:14.175566 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 23:57:14.175611 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":114,"threads_started":1}
I20260806 23:57:14.175999 16186 raft_consensus.cc:2454] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62a9f6042d548d585f45e196430a1ef in term 0.
I20260806 23:57:14.176182 15995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62a9f6042d548d585f45e196430a1ef" is_pre_election: true
I20260806 23:57:14.176374 15995 raft_consensus.cc:2454] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ace18930734c1e8f0c6ab11c5871ba in term 0.
I20260806 23:57:14.176514 15909 leader_election.cc:302] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4241c3db55754126ab837fab1c79b63f, d62a9f6042d548d585f45e196430a1ef; no voters:
I20260806 23:57:14.176774 16282 raft_consensus.cc:2781] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:14.176783 15726 leader_election.cc:302] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6ace18930734c1e8f0c6ab11c5871ba, d62a9f6042d548d585f45e196430a1ef; no voters:
I20260806 23:57:14.176878 16282 raft_consensus.cc:482] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:14.176932 16282 raft_consensus.cc:3037] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:14.176987 16186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241c3db55754126ab837fab1c79b63f" is_pre_election: true
I20260806 23:57:14.177054 16284 raft_consensus.cc:2781] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:14.177126 16186 raft_consensus.cc:2454] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ace18930734c1e8f0c6ab11c5871ba in term 0.
I20260806 23:57:14.177151 16284 raft_consensus.cc:482] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:14.177201 16284 raft_consensus.cc:3037] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:14.178303 16282 raft_consensus.cc:504] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.178429 16284 raft_consensus.cc:504] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.178680 16282 leader_election.cc:288] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [CANDIDATE]: Term 1 election: Requested vote from peers 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919), a6ace18930734c1e8f0c6ab11c5871ba (127.8.204.65:44827)
I20260806 23:57:14.178759 16284 leader_election.cc:288] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [CANDIDATE]: Term 1 election: Requested vote from peers d62a9f6042d548d585f45e196430a1ef (127.8.204.66:33669), 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919)
I20260806 23:57:14.179047 16186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "d62a9f6042d548d585f45e196430a1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241c3db55754126ab837fab1c79b63f"
I20260806 23:57:14.179172 16186 raft_consensus.cc:3037] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:14.179337 15819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "d62a9f6042d548d585f45e196430a1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ace18930734c1e8f0c6ab11c5871ba"
I20260806 23:57:14.179349 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241c3db55754126ab837fab1c79b63f"
I20260806 23:57:14.179503 15819 raft_consensus.cc:2379] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d62a9f6042d548d585f45e196430a1ef in current term 1: Already voted for candidate a6ace18930734c1e8f0c6ab11c5871ba in this term.
I20260806 23:57:14.179034 15995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5ef4adcd644f4a90d39672e723fd5" candidate_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62a9f6042d548d585f45e196430a1ef"
I20260806 23:57:14.179651 15995 raft_consensus.cc:2379] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a6ace18930734c1e8f0c6ab11c5871ba in current term 1: Already voted for candidate d62a9f6042d548d585f45e196430a1ef in this term.
I20260806 23:57:14.180088 15726 leader_election.cc:302] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6ace18930734c1e8f0c6ab11c5871ba; no voters: 4241c3db55754126ab837fab1c79b63f, d62a9f6042d548d585f45e196430a1ef
I20260806 23:57:14.180258 16284 raft_consensus.cc:2726] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 23:57:14.180483 16186 raft_consensus.cc:2454] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62a9f6042d548d585f45e196430a1ef in term 1.
I20260806 23:57:14.180761 15909 leader_election.cc:302] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4241c3db55754126ab837fab1c79b63f, d62a9f6042d548d585f45e196430a1ef; no voters: a6ace18930734c1e8f0c6ab11c5871ba
I20260806 23:57:14.180977 16282 raft_consensus.cc:2781] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:14.181195 16282 raft_consensus.cc:686] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 LEADER]: Becoming Leader. State: Replica: d62a9f6042d548d585f45e196430a1ef, State: Running, Role: LEADER
I20260806 23:57:14.181509 16282 consensus_queue.cc:237] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } }
I20260806 23:57:14.184583 15622 catalog_manager.cc:5495] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d62a9f6042d548d585f45e196430a1ef (127.8.204.66). New cstate: current_term: 1 leader_uuid: "d62a9f6042d548d585f45e196430a1ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62a9f6042d548d585f45e196430a1ef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:14.196369 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.195126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.8.204.66:33669)
0806 23:57:14.195235 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.195239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.195366 (+ 127us) server_negotiation.cc:407] Connection header received
0806 23:57:14.195483 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.195485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.195533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.195598 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.196110 (+ 512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.196112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.196114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.196124 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.196139 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.196145 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.196208 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.196262 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.196290 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:57:14.201615 15819 raft_consensus.cc:1261] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba [term 1 FOLLOWER]: Refusing update from remote peer d62a9f6042d548d585f45e196430a1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:14.201615 16186 raft_consensus.cc:1261] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 1 FOLLOWER]: Refusing update from remote peer d62a9f6042d548d585f45e196430a1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:14.202126 16291 consensus_queue.cc:1035] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4241c3db55754126ab837fab1c79b63f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:14.202294 16282 consensus_queue.cc:1035] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ace18930734c1e8f0c6ab11c5871ba" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:57:14.207621 15883 tablet.cc:2369] T 9fb5ef4adcd644f4a90d39672e723fd5 P a6ace18930734c1e8f0c6ab11c5871ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:14.209542 16296 mvcc.cc:205] Tried to move back new op lower bound from 7315704357683789824 to 7315704357609955328. Current Snapshot: MvccSnapshot[applied={T|T < 7315704357683789824}]
I20260806 23:57:14.211382 16302 mvcc.cc:205] Tried to move back new op lower bound from 7315704357683789824 to 7315704357609955328. Current Snapshot: MvccSnapshot[applied={T|T < 7315704357683789824}]
I20260806 23:57:14.219219 15617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:14.219341 15617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:14.221427 15617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 074b1d25fdf14738a35d76420248eb5d: Sending DeleteTablet for 3 replicas of tablet 9fb5ef4adcd644f4a90d39672e723fd5
I20260806 23:57:14.222110 15969 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:14 UTC) from {username='slave'} at 127.0.0.1:56594
I20260806 23:57:14.222180 16155 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:14 UTC) from {username='slave'} at 127.0.0.1:54274
I20260806 23:57:14.222313 15786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb5ef4adcd644f4a90d39672e723fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:14 UTC) from {username='slave'} at 127.0.0.1:33052
I20260806 23:57:14.222725 16316 tablet_replica.cc:330] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: stopping tablet replica
I20260806 23:57:14.223038 16315 tablet_replica.cc:330] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: stopping tablet replica
I20260806 23:57:14.223038 16316 raft_consensus.cc:2229] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:14.223095 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 15701
I20260806 23:57:14.225601 16316 raft_consensus.cc:2258] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:14.229542 16315 raft_consensus.cc:2229] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:14.229874 16315 raft_consensus.cc:2258] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:14.231349 16316 ts_tablet_manager.cc:1905] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:14.231564 16315 ts_tablet_manager.cc:1905] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:14.234001 16315 ts_tablet_manager.cc:1918] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:14.234109 16315 log.cc:1196] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/9fb5ef4adcd644f4a90d39672e723fd5
I20260806 23:57:14.234169 16316 ts_tablet_manager.cc:1918] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:14.234257 16316 log.cc:1196] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/9fb5ef4adcd644f4a90d39672e723fd5
I20260806 23:57:14.234472 16315 ts_tablet_manager.cc:1939] T 9fb5ef4adcd644f4a90d39672e723fd5 P d62a9f6042d548d585f45e196430a1ef: Deleting consensus metadata
I20260806 23:57:14.234587 16316 ts_tablet_manager.cc:1939] T 9fb5ef4adcd644f4a90d39672e723fd5 P 4241c3db55754126ab837fab1c79b63f: Deleting consensus metadata
I20260806 23:57:14.235631 15604 catalog_manager.cc:4841] TS d62a9f6042d548d585f45e196430a1ef (127.8.204.66:33669): tablet 9fb5ef4adcd644f4a90d39672e723fd5 (table test-table [id=c4f009342ec34adba55e206b636e6e69]) successfully deleted
I20260806 23:57:14.235685 15602 catalog_manager.cc:4841] TS 4241c3db55754126ab837fab1c79b63f (127.8.204.67:40919): tablet 9fb5ef4adcd644f4a90d39672e723fd5 (table test-table [id=c4f009342ec34adba55e206b636e6e69]) successfully deleted
W20260806 23:57:14.241904 15603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:44827 (error 108)
I20260806 23:57:14.241971 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 15885
I20260806 23:57:14.246172 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.245923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44827 (local address 127.0.0.1:33064)
0806 23:57:14.246045 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.246104 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 23:57:14.246315 15603 catalog_manager.cc:4584] TS a6ace18930734c1e8f0c6ab11c5871ba (127.8.204.65:44827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fb5ef4adcd644f4a90d39672e723fd5: Network error: Client connection negotiation failed: client connection to 127.8.204.65:44827: connect: Connection refused (error 111)
I20260806 23:57:14.250908 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16066
I20260806 23:57:14.259435 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 15577
2026-08-06T23:57:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (961 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 23:57:14.287002 9009 mini_oidc.cc:152] Starting JWKS server
I20260806 23:57:14.288602 9009 webserver.cc:460] Webserver started at https://127.0.0.1:46867/ using document root <none> and password file <none>
I20260806 23:57:14.288789 9009 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:57:14.289031 9009 webserver.cc:460] Webserver started at http://127.0.0.1:42421/ using document root <none> and password file <none>
2026-08-06T23:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:14Z Disabled control of system clock
I20260806 23:57:14.297899 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40189
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46867/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/testchainca.cert with env {}
W20260806 23:57:14.402426 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:14.402658 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:14.402695 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:14.406332 16331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:14.406378 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:14.406391 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:14.406424 16331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:14.406440 16331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:57:14.406467 16331 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:57:14.406479 16331 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46867/jwks.json
I20260806 23:57:14.411175 16331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46867/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:14.412091 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:14.412985 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:14.420680 16337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.420688 16340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.420697 16338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:14.420976 16331 server_base.cc:1034] running on GCE node
I20260806 23:57:14.421586 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:14.422595 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:14.423771 16331 hybrid_clock.cc:648] HybridClock initialized: now 1786060634423758 us; error 25 us; skew 500 ppm
I20260806 23:57:14.425429 16331 webserver.cc:460] Webserver started at http://127.8.204.126:46799/ using document root <none> and password file <none>
I20260806 23:57:14.425956 16331 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:14.426056 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:14.426282 16331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:14.428200 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "b109a2660d374c039e16c962ac330bb8"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "d5b2efa9e3f965841dc5e89e5348a4c2"
server_key_iv: "2dcddedaa04e27dc5ef56c20d81eab30"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.428738 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "b109a2660d374c039e16c962ac330bb8"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "d5b2efa9e3f965841dc5e89e5348a4c2"
server_key_iv: "2dcddedaa04e27dc5ef56c20d81eab30"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.432164 16331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 23:57:14.434430 16347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:14.435571 16331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:14.435704 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "b109a2660d374c039e16c962ac330bb8"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "d5b2efa9e3f965841dc5e89e5348a4c2"
server_key_iv: "2dcddedaa04e27dc5ef56c20d81eab30"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.435801 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:14.454895 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:14.455567 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:14.455741 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:14.462504 16331 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40189
I20260806 23:57:14.462515 16415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40189 every 8 connection(s)
I20260806 23:57:14.463429 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:14.466053 16416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:14.471321 16416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8: Bootstrap starting.
I20260806 23:57:14.473388 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16331
I20260806 23:57:14.473542 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:14.473834 9009 external_mini_cluster.cc:1428] Setting key ff98c583c9d34fae37efc2b479628ee8
I20260806 23:57:14.473920 16416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:14.474778 16416 log.cc:824] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:14.476624 16416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8: No bootstrap required, opened a new log
I20260806 23:57:14.479739 16416 raft_consensus.cc:348] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } }
I20260806 23:57:14.479924 16416 raft_consensus.cc:374] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:14.480002 16416 raft_consensus.cc:729] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b109a2660d374c039e16c962ac330bb8, State: Initialized, Role: FOLLOWER
I20260806 23:57:14.480450 16416 consensus_queue.cc:260] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } }
I20260806 23:57:14.480597 16416 raft_consensus.cc:388] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:14.480671 16416 raft_consensus.cc:482] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:14.480759 16416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:14.481900 16416 raft_consensus.cc:504] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } }
I20260806 23:57:14.482303 16416 leader_election.cc:302] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b109a2660d374c039e16c962ac330bb8; no voters:
I20260806 23:57:14.482453 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.474906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.8.204.126:40189)
0806 23:57:14.475641 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.475660 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.475699 (+ 39us) server_negotiation.cc:407] Connection header received
0806 23:57:14.476266 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.476281 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.476563 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.476877 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.477260 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.478117 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.478983 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.479402 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.479569 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.480335 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.480340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.480354 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.480802 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.480869 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.480896 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.481052 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.481171 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.481430 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":38144,"thread_start_us":141,"threads_started":1}
I20260806 23:57:14.482621 16416 leader_election.cc:288] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:14.482851 16422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:14.483057 16422 raft_consensus.cc:686] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [term 1 LEADER]: Becoming Leader. State: Replica: b109a2660d374c039e16c962ac330bb8, State: Running, Role: LEADER
I20260806 23:57:14.483328 16422 consensus_queue.cc:237] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } }
I20260806 23:57:14.483790 16416 sys_catalog.cc:564] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:14.484475 16425 sys_catalog.cc:455] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b109a2660d374c039e16c962ac330bb8. Latest consensus state: current_term: 1 leader_uuid: "b109a2660d374c039e16c962ac330bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } } }
I20260806 23:57:14.484618 16425 sys_catalog.cc:458] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:14.484854 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b109a2660d374c039e16c962ac330bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b109a2660d374c039e16c962ac330bb8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40189 } } }
I20260806 23:57:14.484930 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:14.485025 16433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:14.485234 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:57:14.488019 16433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:14.493256 16433 catalog_manager.cc:1347] Generated new cluster ID: 66993a6d153349bf93e16db881f7a413
I20260806 23:57:14.493317 16433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:14.500000 16433 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:14.500808 16433 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:14.513742 16433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b109a2660d374c039e16c962ac330bb8: Generated new TSK 0
I20260806 23:57:14.514348 16433 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 23:57:14.547777 16455 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 23:57:14.547911 16455 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 23:57:14.548410 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.525002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.8.204.126:40189)
0806 23:57:14.525140 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.525146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.525342 (+ 196us) server_negotiation.cc:407] Connection header received
0806 23:57:14.525566 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.525569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.525627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.525738 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:57:14.525994 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.526473 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.527349 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.527584 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.527687 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.528017 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:57:14.547992 (+ 19975us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 23:57:14.548283 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 23:57:14.548481 16421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 23:57:14.549595 16452 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:14.524938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40189 (local address 127.0.0.1:39788)
0806 23:57:14.525263 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.525275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.525414 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.525756 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.525759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.525767 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:57:14.525892 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:14.525897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.526578 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.526582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:14.527232 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:14.527241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.527705 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.527709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:14.527803 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.527854 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:57:14.548218 (+ 20364us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 23:57:14.548426 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.126:40189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.204.126:40189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 23:57:14.550186 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16331
2026-08-06T23:57:14Z chronyd exiting
I20260806 23:57:14.594251 9009 test_util.cc:175] -----------------------------------------------
I20260806 23:57:14.594321 9009 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060609240246-9009-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 23:57:14.595407 9009 mini_oidc.cc:152] Starting JWKS server
I20260806 23:57:14.596180 9009 webserver.cc:460] Webserver started at https://127.0.0.1:33739/ using document root <none> and password file <none>
I20260806 23:57:14.596282 9009 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:57:14.596521 9009 webserver.cc:460] Webserver started at http://127.0.0.1:45143/ using document root <none> and password file <none>
2026-08-06T23:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:14Z Disabled control of system clock
I20260806 23:57:14.605607 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:34875
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:34875
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/testchainca.cert with env {}
W20260806 23:57:14.709170 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:14.709381 16467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:14.709431 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:14.713263 16467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:14.713320 16467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:14.713335 16467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:14.713375 16467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:14.713405 16467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:57:14.713446 16467 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:57:14.713483 16467 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33739/jwks.json
I20260806 23:57:14.718616 16467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:34875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:14.719554 16467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:14.720463 16467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:14.728319 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.728324 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:14.728332 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:14.728514 16467 server_base.cc:1034] running on GCE node
I20260806 23:57:14.729068 16467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:14.730108 16467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:14.731266 16467 hybrid_clock.cc:648] HybridClock initialized: now 1786060634731237 us; error 39 us; skew 500 ppm
I20260806 23:57:14.732964 16467 webserver.cc:460] Webserver started at http://127.8.204.126:43753/ using document root <none> and password file <none>
I20260806 23:57:14.733484 16467 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:14.733538 16467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:14.733750 16467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:14.735642 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "49609f28a7454ab987801de112372786"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "6ff7ef840672a579728e1161741e1d6a"
server_key_iv: "0fbe44f0e11c2a64f2a3c46ee53631c1"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.736212 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "49609f28a7454ab987801de112372786"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "6ff7ef840672a579728e1161741e1d6a"
server_key_iv: "0fbe44f0e11c2a64f2a3c46ee53631c1"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.739681 16467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 23:57:14.741855 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:14.743014 16467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:14.743132 16467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "49609f28a7454ab987801de112372786"
format_stamp: "Formatted at 2026-08-06 23:57:14 on dist-test-slave-39l8"
server_key: "6ff7ef840672a579728e1161741e1d6a"
server_key_iv: "0fbe44f0e11c2a64f2a3c46ee53631c1"
server_key_version: "encryptionkey@0"
I20260806 23:57:14.743249 16467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:14.757058 16467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:14.757588 16467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:14.757750 16467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:14.764282 16467 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:34875
I20260806 23:57:14.764287 16558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:34875 every 8 connection(s)
I20260806 23:57:14.765110 16467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:14.767935 16559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:14.770541 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16467
I20260806 23:57:14.770677 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:14.770954 9009 external_mini_cluster.cc:1428] Setting key 45ddc5ae2c588f5358a43b4b5e343740
I20260806 23:57:14.773168 16559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786: Bootstrap starting.
I20260806 23:57:14.775283 16559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:14.776114 16559 log.cc:824] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:14.777982 16559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786: No bootstrap required, opened a new log
I20260806 23:57:14.779434 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.772257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.8.204.126:34875)
0806 23:57:14.772818 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.772833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.772865 (+ 32us) server_negotiation.cc:407] Connection header received
0806 23:57:14.773488 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.773504 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.773832 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.774066 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:14.774427 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.775313 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.776450 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.776992 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.777100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.777858 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:14.777862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:14.777874 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:14.778241 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:14.778288 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:14.778305 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:14.778374 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:14.778522 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 23:57:14.778792 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260806 23:57:14.781482 16559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } }
I20260806 23:57:14.781669 16559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:14.781744 16559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49609f28a7454ab987801de112372786, State: Initialized, Role: FOLLOWER
I20260806 23:57:14.781958 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:57:14.782187 16559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } }
I20260806 23:57:14.782353 16559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:14.782416 16559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:14.782517 16559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:14.783417 16559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } }
I20260806 23:57:14.783753 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49609f28a7454ab987801de112372786; no voters:
I20260806 23:57:14.783998 16559 leader_election.cc:288] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:14.784066 16565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:14.784334 16565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [term 1 LEADER]: Becoming Leader. State: Replica: 49609f28a7454ab987801de112372786, State: Running, Role: LEADER
I20260806 23:57:14.784603 16565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } }
I20260806 23:57:14.785038 16559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:14.786115 16566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49609f28a7454ab987801de112372786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } } }
I20260806 23:57:14.786139 16568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49609f28a7454ab987801de112372786. Latest consensus state: current_term: 1 leader_uuid: "49609f28a7454ab987801de112372786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49609f28a7454ab987801de112372786" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 34875 } } }
I20260806 23:57:14.786263 16568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:14.786262 16566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:14.786682 16579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:14.789109 16579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:14.793915 16579 catalog_manager.cc:1347] Generated new cluster ID: ed7135f9865346dfad77ba8a842f8226
I20260806 23:57:14.793977 16579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:14.815104 16579 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:14.816082 16579 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:14.822044 16579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49609f28a7454ab987801de112372786: Generated new TSK 0
I20260806 23:57:14.822621 16579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 23:57:14.845592 16602 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 23:57:14.845676 16602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 23:57:14.846199 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:14.831241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.8.204.126:34875)
0806 23:57:14.831361 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:14.831365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:14.831616 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:57:14.831755 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:14.831757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:14.831800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:14.831879 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:57:14.832136 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.832640 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.833774 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.833963 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.834030 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.834327 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:57:14.845895 (+ 11568us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 23:57:14.846105 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 23:57:14.846259 16562 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 23:57:14.846477 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:14.831167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:34875 (local address 127.0.0.1:55984)
0806 23:57:14.831515 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:14.831529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:14.831646 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:14.831899 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:14.831902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:14.831910 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:57:14.832029 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:14.832035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.832748 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.832751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:14.833672 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:14.833682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:14.834060 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:14.834063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:14.834146 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:14.834175 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:57:14.846025 (+ 11850us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 23:57:14.846196 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.126:34875: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260806 23:57:14.846894 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16467
2026-08-06T23:57:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 23:57:14.891422 9009 mini_oidc.cc:152] Starting JWKS server
I20260806 23:57:14.892237 9009 webserver.cc:460] Webserver started at https://127.0.0.1:40437/ using document root <none> and password file <none>
I20260806 23:57:14.892318 9009 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:57:14.892526 9009 webserver.cc:460] Webserver started at http://127.0.0.1:36005/ using document root <none> and password file <none>
2026-08-06T23:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:14Z Disabled control of system clock
I20260806 23:57:14.901926 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38925
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38925
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40437/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 23:57:15.005628 16609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:15.005918 16609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:15.005967 16609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:15.009572 16609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:15.009613 16609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:15.009629 16609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:15.009660 16609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:15.009675 16609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:57:15.009689 16609 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:57:15.009703 16609 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40437/jwks.json
I20260806 23:57:15.014132 16609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40437/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:15.014998 16609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:15.016227 16609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:15.023878 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:15.024137 16609 server_base.cc:1034] running on GCE node
W20260806 23:57:15.023883 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:15.023886 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:15.025537 16609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:15.026540 16609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:15.027719 16609 hybrid_clock.cc:648] HybridClock initialized: now 1786060635027711 us; error 49 us; skew 500 ppm
I20260806 23:57:15.029372 16609 webserver.cc:460] Webserver started at http://127.8.204.126:42019/ using document root <none> and password file <none>
I20260806 23:57:15.029942 16609 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:15.030030 16609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:15.030257 16609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:15.032227 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "ac4d24b2d171425d8de29103fd91c952"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "1120db090781d03d6c0e8d4299b625d1"
server_key_iv: "9bd39f0a991916f23fc3259c661525d6"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.032778 16609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "ac4d24b2d171425d8de29103fd91c952"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "1120db090781d03d6c0e8d4299b625d1"
server_key_iv: "9bd39f0a991916f23fc3259c661525d6"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.036233 16609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:15.038305 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:15.039443 16609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:15.039572 16609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "ac4d24b2d171425d8de29103fd91c952"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "1120db090781d03d6c0e8d4299b625d1"
server_key_iv: "9bd39f0a991916f23fc3259c661525d6"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.039675 16609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:15.053872 16609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:15.054437 16609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:15.054605 16609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:15.061223 16609 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:38925
I20260806 23:57:15.061262 16707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:38925 every 8 connection(s)
I20260806 23:57:15.062098 16609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:15.064741 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:15.066792 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16609
I20260806 23:57:15.066917 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:15.067210 9009 external_mini_cluster.cc:1428] Setting key 3b0af1232dabfa174624a768b39c0ffb
I20260806 23:57:15.070493 16708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952: Bootstrap starting.
I20260806 23:57:15.072640 16708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:15.073350 16708 log.cc:824] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:15.075567 16708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952: No bootstrap required, opened a new log
I20260806 23:57:15.076058 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:15.068514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.8.204.126:38925)
0806 23:57:15.069136 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:15.069151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:15.069184 (+ 33us) server_negotiation.cc:407] Connection header received
0806 23:57:15.069779 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:15.069795 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:15.070140 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:15.070434 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:15.070888 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.071825 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.072728 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.073168 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.073328 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.074173 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:15.074179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:15.074205 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:15.074648 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:15.074742 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:15.074771 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:15.074882 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:15.075004 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 23:57:15.075272 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260806 23:57:15.078758 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:57:15.079532 16708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } }
I20260806 23:57:15.079787 16708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:15.079862 16708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4d24b2d171425d8de29103fd91c952, State: Initialized, Role: FOLLOWER
I20260806 23:57:15.080291 16708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } }
I20260806 23:57:15.080451 16708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:15.080516 16708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:15.080610 16708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:15.081426 16708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } }
I20260806 23:57:15.081755 16708 leader_election.cc:302] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4d24b2d171425d8de29103fd91c952; no voters:
I20260806 23:57:15.082058 16708 leader_election.cc:288] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:15.082157 16713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:15.082388 16713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [term 1 LEADER]: Becoming Leader. State: Replica: ac4d24b2d171425d8de29103fd91c952, State: Running, Role: LEADER
I20260806 23:57:15.082687 16713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } }
I20260806 23:57:15.083207 16708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:15.084390 16716 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4d24b2d171425d8de29103fd91c952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } } }
I20260806 23:57:15.084553 16716 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:15.084389 16717 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4d24b2d171425d8de29103fd91c952. Latest consensus state: current_term: 1 leader_uuid: "ac4d24b2d171425d8de29103fd91c952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4d24b2d171425d8de29103fd91c952" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38925 } } }
I20260806 23:57:15.084802 16717 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:15.085265 16725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:15.087734 16725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:15.092339 16725 catalog_manager.cc:1347] Generated new cluster ID: 2a53f9de949a4f86bcbbf47363017ed7
I20260806 23:57:15.092398 16725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:15.096660 16725 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:15.097429 16725 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:15.103641 16725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4d24b2d171425d8de29103fd91c952: Generated new TSK 0
I20260806 23:57:15.104223 16725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:15.124518 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:15.119826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.8.204.126:38925)
0806 23:57:15.119954 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:15.119958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:15.120138 (+ 180us) server_negotiation.cc:407] Connection header received
0806 23:57:15.120304 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:15.120306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:15.120348 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:15.120429 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:57:15.120711 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.121199 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.122429 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.122626 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.122694 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.123001 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:57:15.124182 (+ 1181us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 23:57:15.124394 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 23:57:15.124586 16711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 23:57:15.126320 16749 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:15.119771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38925 (local address 127.0.0.1:60648)
0806 23:57:15.120046 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:15.120061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:15.120197 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:15.120455 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:15.120458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:15.120483 (+ 25us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:57:15.120597 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:15.120615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.121311 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.121314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:15.122309 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:15.122319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.122752 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.122756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:15.122839 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.122865 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:57:15.124345 (+ 1480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 23:57:15.126152 (+ 1807us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.126:38925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1621,"thread_start_us":103,"threads_started":1}
I20260806 23:57:15.127113 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16609
E20260806 23:57:15.138453 16750 webserver.cc:533] Webserver: sslize failed:
E20260806 23:57:15.138535 16750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T23:57:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (279 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T23:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:15Z Disabled control of system clock
I20260806 23:57:15.180528 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:46551
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:46551
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/testchainca.cert with env {}
W20260806 23:57:15.286449 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:15.286664 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:15.286720 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:15.290361 16758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:15.290412 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:15.290426 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:15.290441 16758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:15.290473 16758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:57:15.290488 16758 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:57:15.290501 16758 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 23:57:15.295293 16758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:46551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:15.296110 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:15.296931 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:15.304440 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:15.304426 16767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:15.304426 16768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:15.304570 16758 server_base.cc:1034] running on GCE node
I20260806 23:57:15.305244 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:15.306143 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:15.307291 16758 hybrid_clock.cc:648] HybridClock initialized: now 1786060635307274 us; error 27 us; skew 500 ppm
I20260806 23:57:15.311529 16758 webserver.cc:460] Webserver started at http://127.8.204.126:39129/ using document root <none> and password file <none>
I20260806 23:57:15.312063 16758 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:15.312130 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:15.312358 16758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:15.314123 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "69f72b953b474666864c8f81e1211dd9"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "8cd957299585150000809e72201b090c"
server_key_iv: "11bd346f9cb84b90cf428f2440ef375b"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.314635 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "69f72b953b474666864c8f81e1211dd9"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "8cd957299585150000809e72201b090c"
server_key_iv: "11bd346f9cb84b90cf428f2440ef375b"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.317983 16758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:15.320108 16778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:15.321146 16758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:15.321262 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "69f72b953b474666864c8f81e1211dd9"
format_stamp: "Formatted at 2026-08-06 23:57:15 on dist-test-slave-39l8"
server_key: "8cd957299585150000809e72201b090c"
server_key_iv: "11bd346f9cb84b90cf428f2440ef375b"
server_key_version: "encryptionkey@0"
I20260806 23:57:15.321341 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:15.346665 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:15.347244 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:15.347368 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:15.353955 16758 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:46551
I20260806 23:57:15.353965 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:46551 every 8 connection(s)
I20260806 23:57:15.354756 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:15.356320 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16758
I20260806 23:57:15.356446 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:15.356737 9009 external_mini_cluster.cc:1428] Setting key a6f37d03bfaf3f2a2aaab4580a312326
I20260806 23:57:15.357983 16851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:15.364241 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9: Bootstrap starting.
I20260806 23:57:15.365751 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:15.358057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.8.204.126:46551)
0806 23:57:15.358590 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:15.358604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:15.358635 (+ 31us) server_negotiation.cc:407] Connection header received
0806 23:57:15.359259 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:15.359275 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:15.359616 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:15.359868 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:15.360365 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.361550 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.362769 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.363216 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.363358 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.364163 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:15.364168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:15.364180 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:15.364562 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:15.364614 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:15.364647 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:15.364721 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:15.364869 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 23:57:15.365134 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":113,"threads_started":1}
I20260806 23:57:15.366420 16851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:15.367252 16851 log.cc:824] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:15.368386 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:57:15.369146 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9: No bootstrap required, opened a new log
I20260806 23:57:15.372606 16851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } }
I20260806 23:57:15.372797 16851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:15.372871 16851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f72b953b474666864c8f81e1211dd9, State: Initialized, Role: FOLLOWER
I20260806 23:57:15.373319 16851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } }
I20260806 23:57:15.373451 16851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:15.373517 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:15.373620 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:15.374457 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } }
I20260806 23:57:15.374835 16851 leader_election.cc:302] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f72b953b474666864c8f81e1211dd9; no voters:
I20260806 23:57:15.375160 16851 leader_election.cc:288] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:15.375265 16857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:15.375490 16857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 69f72b953b474666864c8f81e1211dd9, State: Running, Role: LEADER
I20260806 23:57:15.375808 16857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } }
I20260806 23:57:15.376248 16851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:15.377426 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f72b953b474666864c8f81e1211dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } } }
I20260806 23:57:15.377574 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:15.377874 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f72b953b474666864c8f81e1211dd9. Latest consensus state: current_term: 1 leader_uuid: "69f72b953b474666864c8f81e1211dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f72b953b474666864c8f81e1211dd9" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46551 } } }
I20260806 23:57:15.377965 16869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:15.377985 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:15.380663 16869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:15.386161 16869 catalog_manager.cc:1347] Generated new cluster ID: 37d2cfe4894f430e942b09498db4a378
I20260806 23:57:15.386269 16869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:15.396839 16869 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:15.397831 16869 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:15.412919 16869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f72b953b474666864c8f81e1211dd9: Generated new TSK 0
I20260806 23:57:15.413547 16869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:15.485947 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:15.424963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.8.204.126:46551)
0806 23:57:15.425069 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:15.425074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:15.425354 (+ 280us) server_negotiation.cc:407] Connection header received
0806 23:57:15.425529 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:15.425542 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:15.425596 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:15.425692 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:57:15.425897 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.426634 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.427451 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.427772 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.427866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.428132 (+ 266us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:57:15.485484 (+ 57352us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 23:57:15.485803 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54596: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 23:57:15.486035 16855 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54596: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 23:57:15.486073 16888 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:15.424889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:46551 (local address 127.0.0.1:54596)
0806 23:57:15.425239 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:15.425251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:15.425379 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:15.425701 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:15.425715 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:15.425721 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:57:15.425805 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:15.425809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.426766 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.426769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:15.427330 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:15.427337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:15.427889 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:15.427892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:15.427955 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:15.427976 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:57:15.485737 (+ 57761us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 23:57:15.485896 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.126:46551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260806 23:57:15.486982 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16758
2026-08-06T23:57:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (341 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:15 dist-test-slave-39l8 krb5kdc[16902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:15 dist-test-slave-39l8 krb5kdc[16902](info): set up 2 sockets
Aug 06 23:57:15 dist-test-slave-39l8 krb5kdc[16902](info): commencing operation
krb5kdc: starting...
W20260806 23:57:16.845162 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:16 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:16.958012 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45969
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:45969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:17.086208 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:17.086504 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:17.086599 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:17.091367 16919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:17.091434 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:17.091470 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:17.091526 16919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:17.091563 16919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:17.097654 16919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:45969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:17.098666 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:17.099655 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:17.108352 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.108441 16931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.108402 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:17.108629 16919 server_base.cc:1034] running on GCE node
I20260806 23:57:17.109293 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:17.110324 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:17.111505 16919 hybrid_clock.cc:648] HybridClock initialized: now 1786060637111481 us; error 27 us; skew 500 ppm
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.114595 16919 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:17.115571 16919 webserver.cc:460] Webserver started at http://127.8.204.126:40191/ using document root <none> and password file <none>
I20260806 23:57:17.116181 16919 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:17.116241 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:17.116449 16919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:17.119316 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "bfeebbd8182445eda9b0feb0c9294b79"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d317dd9e740bb67b0db5cf836689b20e"
server_key_iv: "f725de8761e4907e1f869e16e67f9130"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.120113 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "bfeebbd8182445eda9b0feb0c9294b79"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d317dd9e740bb67b0db5cf836689b20e"
server_key_iv: "f725de8761e4907e1f869e16e67f9130"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.124683 16919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 23:57:17.126986 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.128103 16919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 23:57:17.128233 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "bfeebbd8182445eda9b0feb0c9294b79"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d317dd9e740bb67b0db5cf836689b20e"
server_key_iv: "f725de8761e4907e1f869e16e67f9130"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.128332 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:17.141076 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:17.143677 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:17.143863 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:17.150342 16919 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:45969
I20260806 23:57:17.150364 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:45969 every 8 connection(s)
I20260806 23:57:17.151186 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:17.153707 17015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:17.154139 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 16919
I20260806 23:57:17.154227 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:17.154474 9009 external_mini_cluster.cc:1428] Setting key f93df7b45e219c51279fe5a94ca39824
I20260806 23:57:17.160759 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79: Bootstrap starting.
I20260806 23:57:17.163708 17015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79: Neither blocks nor log segments found. Creating new log.
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:17.164726 17015 log.cc:824] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:17.167063 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79: No bootstrap required, opened a new log
I20260806 23:57:17.170114 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.155856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.8.204.126:45969)
0806 23:57:17.156251 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.156268 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.156304 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:57:17.156954 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.156975 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.157335 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.157666 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.158256 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.159670 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.160794 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.161438 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.161590 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.164793 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.164801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.164816 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.164848 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.166678 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.167168 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.167173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.167178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.167297 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.167606 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.167609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.167610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.168033 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.168204 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.168897 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.169226 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260806 23:57:17.171867 17015 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } }
I20260806 23:57:17.172258 17015 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:17.172374 17015 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfeebbd8182445eda9b0feb0c9294b79, State: Initialized, Role: FOLLOWER
I20260806 23:57:17.172899 17015 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } }
I20260806 23:57:17.173153 17015 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:17.173285 17015 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:17.173437 17015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:17.174759 17015 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } }
I20260806 23:57:17.175192 17015 leader_election.cc:302] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfeebbd8182445eda9b0feb0c9294b79; no voters:
I20260806 23:57:17.175590 17015 leader_election.cc:288] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:17.175698 17023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:17.175961 17023 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [term 1 LEADER]: Becoming Leader. State: Replica: bfeebbd8182445eda9b0feb0c9294b79, State: Running, Role: LEADER
I20260806 23:57:17.176330 17023 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } }
I20260806 23:57:17.176996 17015 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:17.178411 17024 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfeebbd8182445eda9b0feb0c9294b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } } }
I20260806 23:57:17.178545 17024 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:17.178915 17025 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfeebbd8182445eda9b0feb0c9294b79. Latest consensus state: current_term: 1 leader_uuid: "bfeebbd8182445eda9b0feb0c9294b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfeebbd8182445eda9b0feb0c9294b79" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45969 } } }
I20260806 23:57:17.179075 17025 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:17.179436 17038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:17.181953 17038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:17.187080 17038 catalog_manager.cc:1347] Generated new cluster ID: 23c38d20ee4f4cce8a3616aeb4cec853
I20260806 23:57:17.187175 17038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:17.212402 17038 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:17.213543 17038 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:17.227735 17038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfeebbd8182445eda9b0feb0c9294b79: Generated new TSK 0
I20260806 23:57:17.228381 17038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:17.259282 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:17.366451 17048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:17.366676 17048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:17.366763 17048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:17.370520 17048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:17.370566 17048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:17.370659 17048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:17.375638 17048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:17.376567 17048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:17.377635 17048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:17.388072 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.388152 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.388288 17057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:17.388736 17048 server_base.cc:1034] running on GCE node
I20260806 23:57:17.389281 17048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:17.390030 17048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:17.391249 17048 hybrid_clock.cc:648] HybridClock initialized: now 1786060637391222 us; error 35 us; skew 500 ppm
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.396593 17048 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:17.398029 17048 webserver.cc:460] Webserver started at http://127.8.204.65:44979/ using document root <none> and password file <none>
I20260806 23:57:17.398546 17048 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:17.398603 17048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:17.398823 17048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:17.401618 17048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "6f6280b38b6b45ccb4a2ee50a193872f"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "301772bb5a3cd737945c3e6e237283c7"
server_key_iv: "2a31a3e6c7dd78e504e10ec799922218"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.402376 17048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6280b38b6b45ccb4a2ee50a193872f"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "301772bb5a3cd737945c3e6e237283c7"
server_key_iv: "2a31a3e6c7dd78e504e10ec799922218"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.408903 17048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 23:57:17.411897 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.413489 17048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:17.413609 17048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "6f6280b38b6b45ccb4a2ee50a193872f"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "301772bb5a3cd737945c3e6e237283c7"
server_key_iv: "2a31a3e6c7dd78e504e10ec799922218"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.413722 17048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:17.436785 17048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:17.439998 17048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:17.440201 17048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:17.440747 17048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:17.441571 17048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:17.441641 17048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.441710 17048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:17.441757 17048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.450044 17048 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:33309
I20260806 23:57:17.450073 17215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:33309 every 8 connection(s)
I20260806 23:57:17.450975 17048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:17.455118 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17048
I20260806 23:57:17.455243 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:17.455565 9009 external_mini_cluster.cc:1428] Setting key 1a3d58917016fd1dbe7614440958a9ed
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:17.464751 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.452716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:52429 (local address 127.8.204.126:45969)
0806 23:57:17.452897 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.452901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.453688 (+ 787us) server_negotiation.cc:407] Connection header received
0806 23:57:17.455101 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.455104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.455167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.455285 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.456611 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.457122 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.458193 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.458406 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.458504 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.460924 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.460930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.460936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.460967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.462670 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.463155 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.463162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.463163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.463262 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.463621 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.463626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.463629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.463886 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.464006 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.464428 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.464557 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 23:57:17.465329 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.453113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45969 (local address 127.8.204.65:52429)
0806 23:57:17.453540 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:17.453587 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:17.454875 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:17.455585 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:17.455596 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:17.456035 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:17.456470 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.456483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.457257 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.457260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.458064 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.458085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.458541 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.458545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.458695 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.459248 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:17.459263 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:17.460764 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:17.462805 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.462810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.462820 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.463051 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.463360 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.463362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.463363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.463497 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.463991 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:17.463996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:17.464192 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 23:57:17.464334 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 23:57:17.464545 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:57:17.466435 17216 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45969
I20260806 23:57:17.466658 17216 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:17.467136 17216 heartbeater.cc:511] Master 127.8.204.126:45969 requested a full tablet report, sending...
I20260806 23:57:17.468780 16966 ts_manager.cc:194] Registered new tserver with Master: 6f6280b38b6b45ccb4a2ee50a193872f (127.8.204.65:33309)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260806 23:57:17.470397 16966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:52429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:17.483554 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:17.589272 17223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:17.589481 17223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:17.589519 17223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:17.593268 17223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:17.593312 17223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:17.593376 17223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:17.597955 17223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:17.598842 17223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:17.599716 17223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:17.608554 17232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.608551 17229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.608551 17230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:17.608717 17223 server_base.cc:1034] running on GCE node
I20260806 23:57:17.609187 17223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:17.609753 17223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:17.610924 17223 hybrid_clock.cc:648] HybridClock initialized: now 1786060637610900 us; error 43 us; skew 500 ppm
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.613461 17223 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:17.614264 17223 webserver.cc:460] Webserver started at http://127.8.204.66:41347/ using document root <none> and password file <none>
I20260806 23:57:17.614823 17223 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:17.614910 17223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:17.615129 17223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:17.616927 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "77f52270dfc14df6a264c3b3295f058b"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "fe27cf49ca4a5d8c12531e5d06789ec3"
server_key_iv: "f094788ea440c98cc9dd769724cea74f"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.617642 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "77f52270dfc14df6a264c3b3295f058b"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "fe27cf49ca4a5d8c12531e5d06789ec3"
server_key_iv: "f094788ea440c98cc9dd769724cea74f"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.621554 17223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:57:17.623880 17242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.624872 17223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:17.625015 17223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "77f52270dfc14df6a264c3b3295f058b"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "fe27cf49ca4a5d8c12531e5d06789ec3"
server_key_iv: "f094788ea440c98cc9dd769724cea74f"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.625115 17223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:17.641762 17223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:17.644152 17223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:17.644323 17223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:17.644811 17223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:17.645533 17223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:17.645596 17223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.645656 17223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:17.645700 17223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.654165 17223 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:37423
I20260806 23:57:17.654191 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:37423 every 8 connection(s)
I20260806 23:57:17.654983 17223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:17.659787 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17223
I20260806 23:57:17.659912 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:17.660233 9009 external_mini_cluster.cc:1428] Setting key d40de563e06077a6387934772c52b4e9
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:17.668152 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.656565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:50033 (local address 127.8.204.126:45969)
0806 23:57:17.656703 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.656708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.657516 (+ 808us) server_negotiation.cc:407] Connection header received
0806 23:57:17.658762 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.658765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.658821 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.658891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.660127 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.660677 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.662004 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.662233 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.662343 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.664778 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.664782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.664785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.664814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.666187 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.666668 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.666673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.666675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.666784 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.667131 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.667135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.667137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.667312 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.667413 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.667869 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.667983 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:57:17.668819 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.656943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45969 (local address 127.8.204.66:50033)
0806 23:57:17.657365 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:17.657405 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:17.658544 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:17.659026 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:17.659032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:17.659359 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:17.659974 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.659994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.660800 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.660804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.661878 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.661889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.662380 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.662384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.662632 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.663242 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:17.663258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:17.664622 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:17.666288 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.666294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.666306 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.666557 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.666885 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.666888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.666890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.667030 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.667427 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:17.667435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:17.667656 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 23:57:17.667822 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 23:57:17.668101 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260806 23:57:17.670027 17390 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45969
I20260806 23:57:17.670282 17390 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:57:17.670975 17390 heartbeater.cc:511] Master 127.8.204.126:45969 requested a full tablet report, sending...
I20260806 23:57:17.672058 16966 ts_manager.cc:194] Registered new tserver with Master: 77f52270dfc14df6a264c3b3295f058b (127.8.204.66:37423)
I20260806 23:57:17.672628 16966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:50033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:17.687013 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:17.793030 17395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:17.793287 17395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:17.793368 17395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:17.796941 17395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:17.797012 17395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:17.797109 17395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:17.801898 17395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:17.802881 17395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:17.803822 17395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:17.812893 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.812940 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:17.812904 17406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:17.813084 17395 server_base.cc:1034] running on GCE node
I20260806 23:57:17.813632 17395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:17.814319 17395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:17.815536 17395 hybrid_clock.cc:648] HybridClock initialized: now 1786060637815514 us; error 34 us; skew 500 ppm
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.818163 17395 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:17.819036 17395 webserver.cc:460] Webserver started at http://127.8.204.67:33647/ using document root <none> and password file <none>
I20260806 23:57:17.819537 17395 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:17.819625 17395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:17.819815 17395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:17.821739 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "17bee446504741b499dbb9bf030176b5"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d9c1a6ffe1440f383d43192f3f30eb19"
server_key_iv: "03f176f469daf3ad7c4949879d58aa33"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.822314 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "17bee446504741b499dbb9bf030176b5"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d9c1a6ffe1440f383d43192f3f30eb19"
server_key_iv: "03f176f469daf3ad7c4949879d58aa33"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.825627 17395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:57:17.827920 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.829077 17395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:17.829205 17395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "17bee446504741b499dbb9bf030176b5"
format_stamp: "Formatted at 2026-08-06 23:57:17 on dist-test-slave-39l8"
server_key: "d9c1a6ffe1440f383d43192f3f30eb19"
server_key_iv: "03f176f469daf3ad7c4949879d58aa33"
server_key_version: "encryptionkey@0"
I20260806 23:57:17.829303 17395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:17.862470 17395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:17.865515 17395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:17.865712 17395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:17.866250 17395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:17.867110 17395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:17.867174 17395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.867237 17395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:17.867280 17395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:17.876026 17395 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:42279
I20260806 23:57:17.876062 17564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:42279 every 8 connection(s)
I20260806 23:57:17.876986 17395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:17.883308 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17395
I20260806 23:57:17.883455 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:17.883737 9009 external_mini_cluster.cc:1428] Setting key f3eb8cd5cb6e251217693305151ac133
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:17.890147 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.878994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:37789 (local address 127.8.204.126:45969)
0806 23:57:17.879164 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.879169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.880015 (+ 846us) server_negotiation.cc:407] Connection header received
0806 23:57:17.881313 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.881316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.881365 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.881456 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.882541 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.883170 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.884433 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.884736 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.884819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.887312 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.887315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.887317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.887338 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.888502 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.888931 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.888934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.888935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.889004 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.889284 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.889286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.889287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.889441 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.889512 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.889937 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.890023 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:57:17.891002 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.879430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45969 (local address 127.8.204.67:37789)
0806 23:57:17.879844 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:17.879897 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:17.881130 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:17.881616 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:17.881622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:17.881958 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:17.882418 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.882431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.883300 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.883303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.884304 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:17.884315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.884890 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.884896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:17.885065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.885685 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:17.885708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:17.887169 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:17.888602 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.888606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.888616 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.888850 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.889077 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:17.889087 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:17.889088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:17.889210 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:17.889521 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:17.889526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:17.889731 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 23:57:17.889897 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 23:57:17.890140 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260806 23:57:17.892076 17565 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45969
I20260806 23:57:17.892302 17565 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:17.892783 17565 heartbeater.cc:511] Master 127.8.204.126:45969 requested a full tablet report, sending...
I20260806 23:57:17.893721 16966 ts_manager.cc:194] Registered new tserver with Master: 17bee446504741b499dbb9bf030176b5 (127.8.204.67:42279)
I20260806 23:57:17.894199 16966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:37789
I20260806 23:57:17.897243 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/test-admin.keytab.
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.912374 9009 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 23:57:17.914974 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:17.913567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.8.204.65:33309)
0806 23:57:17.913868 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.913882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.913904 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:57:17.913952 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.913956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.914115 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.914231 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.914797 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52608: BlockingRecv error: recv got EOF from 127.0.0.1:52608 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":125,"threads_started":1}
W20260806 23:57:17.914995 9009 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.915922 9009 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.916776 9009 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:57:17.925477 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.917563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.8.204.65:33309)
0806 23:57:17.917684 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.917686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.917723 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:57:17.917814 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.917816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.917848 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.917926 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.918392 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.919248 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.920065 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.920410 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.920489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.922550 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.922559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.922586 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.922608 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.923986 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.924359 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.924363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.924364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.924435 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.924648 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.924650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.924651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.924859 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.925003 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.925177 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.925352 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 23:57:17.928825 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:17.928037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.8.204.65:33309)
0806 23:57:17.928148 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.928151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.928241 (+ 90us) server_negotiation.cc:407] Connection header received
0806 23:57:17.928322 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.928324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.928353 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.928411 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.928741 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52628: BlockingRecv error: recv got EOF from 127.0.0.1:52628 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260806 23:57:17.928975 9009 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.929850 9009 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.930666 9009 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:57:17.938696 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.931427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.8.204.65:33309)
0806 23:57:17.931545 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.931549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.931704 (+ 155us) server_negotiation.cc:407] Connection header received
0806 23:57:17.931792 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.931794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.931824 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.931886 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.932413 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.933017 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.933720 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.934045 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.934135 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.936192 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.936194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.936196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.936217 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.937369 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.937733 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.937736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.937737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.937805 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.938024 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.938026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.938027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.938168 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.938257 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.938433 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.938541 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 23:57:17.941089 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:17.940167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.8.204.65:33309)
0806 23:57:17.940275 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.940278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.940410 (+ 132us) server_negotiation.cc:407] Connection header received
0806 23:57:17.940520 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.940523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.940559 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.940635 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.940997 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52634: BlockingRecv error: recv got EOF from 127.0.0.1:52634 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 23:57:17.941238 9009 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060609240246-9009-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.942183 9009 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:17.943048 9009 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:57:17 dist-test-slave-39l8 krb5kdc[16902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:57:17.950400 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:17.943735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.8.204.65:33309)
0806 23:57:17.943849 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:17.943853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:17.943935 (+ 82us) server_negotiation.cc:407] Connection header received
0806 23:57:17.944062 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:17.944065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:17.944106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:17.944183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:17.944644 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.945092 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.945846 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:17.946044 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:17.946145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:17.947963 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:17.947966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:17.947969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:17.947991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:17.948985 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.949361 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.949365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.949366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.949438 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:17.949690 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:17.949693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:17.949695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:17.949835 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:17.949945 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:17.950171 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 23:57:17.950269 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:57:17.951284 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17048
I20260806 23:57:17.966864 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17223
I20260806 23:57:17.980521 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17395
I20260806 23:57:17.990222 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 16919
2026-08-06T23:57:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2269 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:18 dist-test-slave-39l8 krb5kdc[17593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:18 dist-test-slave-39l8 krb5kdc[17593](info): set up 2 sockets
Aug 06 23:57:18 dist-test-slave-39l8 krb5kdc[17593](info): commencing operation
krb5kdc: starting...
W20260806 23:57:19.118468 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab.
Entry for principal oryx/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab.
I20260806 23:57:19.233513 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38385
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab
--principal=oryx/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:19.339047 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:19.339258 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:19.339296 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:19.343253 17615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:19.343300 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:19.343319 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:19.343351 17615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:19.343381 17615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:19.348281 17615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab
--principal=oryx/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:19.349150 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:19.350054 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:19.358088 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:19.358223 17615 server_base.cc:1034] running on GCE node
W20260806 23:57:19.358067 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:19.358387 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:19.358942 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:19.359862 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:19.361050 17615 hybrid_clock.cc:648] HybridClock initialized: now 1786060639361019 us; error 26 us; skew 500 ppm
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:19.363827 17615 init.cc:377] Logged in from keytab as oryx/127.8.204.126@KRBTEST.COM (short username oryx)
I20260806 23:57:19.364593 17615 webserver.cc:460] Webserver started at http://127.8.204.126:37177/ using document root <none> and password file <none>
I20260806 23:57:19.365070 17615 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:19.365126 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:19.365278 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:19.367175 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "fd49e3b9534842259ff9f18c7fe9a82b"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "9dfbf17a4ca8eaecaa7d33f29dbf496b"
server_key_iv: "860a184b0ed3d29a3384fcad7ff5866e"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.367662 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "fd49e3b9534842259ff9f18c7fe9a82b"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "9dfbf17a4ca8eaecaa7d33f29dbf496b"
server_key_iv: "860a184b0ed3d29a3384fcad7ff5866e"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.371060 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:19.373248 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.374362 17615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:19.374503 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "fd49e3b9534842259ff9f18c7fe9a82b"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "9dfbf17a4ca8eaecaa7d33f29dbf496b"
server_key_iv: "860a184b0ed3d29a3384fcad7ff5866e"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.374614 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:19.405246 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:19.407807 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:19.407984 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:19.414474 17615 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:38385
I20260806 23:57:19.414542 17711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:38385 every 8 connection(s)
I20260806 23:57:19.415359 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:19.417980 17712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:19.418948 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17615
I20260806 23:57:19.419090 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:19.419385 9009 external_mini_cluster.cc:1428] Setting key b7d1db506682c0c6805719d8b7956341
I20260806 23:57:19.423646 17712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Bootstrap starting.
I20260806 23:57:19.425685 17712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:19.426352 17712 log.cc:824] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:19.428294 17712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: No bootstrap required, opened a new log
I20260806 23:57:19.432834 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:19.420617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.8.204.126:38385)
0806 23:57:19.421000 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:19.421018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:19.421050 (+ 32us) server_negotiation.cc:407] Connection header received
0806 23:57:19.421657 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:19.421672 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:19.421961 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:19.422276 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:19.423148 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.424114 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.424924 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.425362 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.425522 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:19.427935 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:19.427941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:19.427955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:19.427983 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:19.429818 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.430259 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.430269 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.430276 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.430429 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.430674 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.430678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.430679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.431148 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:19.431321 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:19.431614 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 23:57:19.431894 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260806 23:57:19.432801 17712 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } }
I20260806 23:57:19.433058 17712 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:19.433128 17712 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd49e3b9534842259ff9f18c7fe9a82b, State: Initialized, Role: FOLLOWER
I20260806 23:57:19.433569 17712 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } }
I20260806 23:57:19.433735 17712 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:19.433835 17712 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:19.433947 17712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:19.434885 17712 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } }
I20260806 23:57:19.435209 17712 leader_election.cc:302] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd49e3b9534842259ff9f18c7fe9a82b; no voters:
I20260806 23:57:19.435459 17712 leader_election.cc:288] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:19.435570 17720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:19.435801 17720 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 LEADER]: Becoming Leader. State: Replica: fd49e3b9534842259ff9f18c7fe9a82b, State: Running, Role: LEADER
I20260806 23:57:19.436170 17720 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } }
I20260806 23:57:19.436707 17712 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:19.437793 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd49e3b9534842259ff9f18c7fe9a82b. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } }
I20260806 23:57:19.437927 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:19.438323 17730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:19.438292 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } }
I20260806 23:57:19.438422 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:19.441195 17730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:19.446352 17730 catalog_manager.cc:1347] Generated new cluster ID: ff2ee083c70e4140a725dd4276fbee83
I20260806 23:57:19.446409 17730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:19.456141 17730 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:19.457118 17730 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:19.468178 17730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Generated new TSK 0
I20260806 23:57:19.468832 17730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
I20260806 23:57:19.498672 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:19.605381 17743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:19.605589 17743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:19.605626 17743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:19.609041 17743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:19.609086 17743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:19.609133 17743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:19.613869 17743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:19.614710 17743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:19.615567 17743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:19.624282 17749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:19.624291 17748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:19.624396 17743 server_base.cc:1034] running on GCE node
W20260806 23:57:19.624356 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:19.624897 17743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:19.625434 17743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:19.626601 17743 hybrid_clock.cc:648] HybridClock initialized: now 1786060639626588 us; error 35 us; skew 500 ppm
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:19.628994 17743 init.cc:377] Logged in from keytab as oryx/127.8.204.65@KRBTEST.COM (short username oryx)
I20260806 23:57:19.629745 17743 webserver.cc:460] Webserver started at http://127.8.204.65:36731/ using document root <none> and password file <none>
I20260806 23:57:19.630228 17743 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:19.630308 17743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:19.630528 17743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:19.632467 17743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "eaa1cf5ed8cd4dc595518036ee5131ed"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "884a216e39e5b574264b997aa8b58c7f"
server_key_iv: "d6ed94b72da522071e6d36381153a703"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.633001 17743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "eaa1cf5ed8cd4dc595518036ee5131ed"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "884a216e39e5b574264b997aa8b58c7f"
server_key_iv: "d6ed94b72da522071e6d36381153a703"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.636344 17743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:19.638559 17761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.639683 17743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:19.639801 17743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "eaa1cf5ed8cd4dc595518036ee5131ed"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "884a216e39e5b574264b997aa8b58c7f"
server_key_iv: "d6ed94b72da522071e6d36381153a703"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.639892 17743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:19.658421 17743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:19.660974 17743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:19.661141 17743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:19.661644 17743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:19.662384 17743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:19.662449 17743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.662513 17743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:19.662559 17743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.670957 17743 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:44059
I20260806 23:57:19.670994 17910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:44059 every 8 connection(s)
I20260806 23:57:19.671717 17743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:19.674537 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17743
I20260806 23:57:19.674669 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:19.674959 9009 external_mini_cluster.cc:1428] Setting key a2600b4413cf9f5e0c61b350829fa655
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:57:19.685427 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:19.673327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:44961 (local address 127.8.204.126:38385)
0806 23:57:19.673479 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:19.673484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:19.674255 (+ 771us) server_negotiation.cc:407] Connection header received
0806 23:57:19.675684 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:19.675687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:19.675726 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:19.675782 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:19.677272 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.678073 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.679271 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.679485 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.679606 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:19.682104 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:19.682108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:19.682111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:19.682138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:19.683566 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.684037 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.684041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.684043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.684119 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.684435 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.684439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.684441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.684597 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:19.684702 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:19.685158 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 23:57:19.685273 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:19.686141 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:19.673632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:44961)
0806 23:57:19.674078 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:19.674125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:19.675495 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:19.675946 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:19.675954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:19.676405 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:19.677053 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:19.677067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.678203 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.678206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:19.679153 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:19.679162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.679640 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.679645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:19.679793 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:19.680295 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:19.680309 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:19.681939 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:19.683693 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:19.683701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:19.683713 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:19.683940 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:19.684219 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:19.684222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:19.684224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:19.684347 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:19.684695 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:19.684701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:19.684951 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 23:57:19.685114 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 23:57:19.685384 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260806 23:57:19.687376 17911 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.66@KRBTEST.COM" created.
I20260806 23:57:19.687631 17911 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:19.688212 17911 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:19.689702 17660 ts_manager.cc:194] Registered new tserver with Master: eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059)
I20260806 23:57:19.690671 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:44961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
I20260806 23:57:19.701287 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:19.805158 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:19.805358 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:19.805394 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:19.808936 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:19.808981 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:19.809029 17918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:19.813673 17918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:19.814450 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:19.815312 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:19.824842 17928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:19.824821 17925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:19.824822 17926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:19.824901 17918 server_base.cc:1034] running on GCE node
I20260806 23:57:19.825502 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:19.826010 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:19.827174 17918 hybrid_clock.cc:648] HybridClock initialized: now 1786060639827162 us; error 24 us; skew 500 ppm
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:19.829547 17918 init.cc:377] Logged in from keytab as oryx/127.8.204.66@KRBTEST.COM (short username oryx)
I20260806 23:57:19.830253 17918 webserver.cc:460] Webserver started at http://127.8.204.66:42661/ using document root <none> and password file <none>
I20260806 23:57:19.830782 17918 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:19.830864 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:19.831065 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:19.832886 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "f0d04faeb4524808804329390eab1ede"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "f672bab5f4db1646229122bff829450b"
server_key_iv: "67606faf3e23845eca5c9b398636ec1c"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.833403 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "f0d04faeb4524808804329390eab1ede"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "f672bab5f4db1646229122bff829450b"
server_key_iv: "67606faf3e23845eca5c9b398636ec1c"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.836740 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:19.838995 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.840072 17918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:19.840201 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "f0d04faeb4524808804329390eab1ede"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "f672bab5f4db1646229122bff829450b"
server_key_iv: "67606faf3e23845eca5c9b398636ec1c"
server_key_version: "encryptionkey@0"
I20260806 23:57:19.840299 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:19.868016 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:19.870522 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:19.870710 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:19.871270 17918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:19.872059 17918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:19.872126 17918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.872202 17918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:19.872246 17918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:19.880906 17918 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44465
I20260806 23:57:19.880936 18100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44465 every 8 connection(s)
I20260806 23:57:19.881685 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:19.887068 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 17918
I20260806 23:57:19.887193 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:19.887487 9009 external_mini_cluster.cc:1428] Setting key dc58909fdef13c6c08bb0895d2036f21
Aug 06 23:57:19 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:19.895167 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:19.883292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45733 (local address 127.8.204.126:38385)
0806 23:57:19.883439 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:19.883443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:19.884174 (+ 731us) server_negotiation.cc:407] Connection header received
0806 23:57:19.885305 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:19.885308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:19.885352 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:19.885441 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:19.886470 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.887197 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.888429 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.888621 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.888725 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:19.891193 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:19.891196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:19.891198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:19.891219 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:19.892636 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.893332 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.893336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.893337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.893424 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:19.893828 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:19.893831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:19.893832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:19.893991 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:19.894068 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:19.894916 (+ 848us) server_negotiation.cc:299] Negotiation successful
0806 23:57:19.895045 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:57:19.896217 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:19.883632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:45733)
0806 23:57:19.884034 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:19.884072 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:19.885133 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:19.885590 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:19.885596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:19.885920 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:19.886350 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:19.886363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.887336 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.887338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:19.888328 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:19.888338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:19.888771 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:19.888774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:19.888948 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:19.889453 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:19.889476 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:19.891032 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:19.892774 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:19.892782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:19.892800 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:19.893153 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:19.893520 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:19.893527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:19.893531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:19.893726 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:19.894095 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:19.894104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:19.894438 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 23:57:19.894653 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 23:57:19.895147 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260806 23:57:19.897382 18101 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
I20260806 23:57:19.897596 18101 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:19.898023 18101 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:19.899072 17660 ts_manager.cc:194] Registered new tserver with Master: f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
WARNING: no policy specified for oryx/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:57:19.899820 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:45733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
I20260806 23:57:19.917060 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:20.031937 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:20.032248 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:20.032289 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:20.036063 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:20.036108 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:20.036157 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:20.040834 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:20.041690 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:20.042647 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:20.051842 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.052012 18117 server_base.cc:1034] running on GCE node
W20260806 23:57:20.051836 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.051854 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.052572 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.053093 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.054283 18117 hybrid_clock.cc:648] HybridClock initialized: now 1786060640054268 us; error 31 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.056957 18117 init.cc:377] Logged in from keytab as oryx/127.8.204.67@KRBTEST.COM (short username oryx)
I20260806 23:57:20.057705 18117 webserver.cc:460] Webserver started at http://127.8.204.67:34471/ using document root <none> and password file <none>
I20260806 23:57:20.058149 18117 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.058208 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.058351 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:20.060272 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "316e1b05a7a04d338b5f8b422927fbfd"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "1ec10cc52380257c4c017bfc318dc529"
server_key_iv: "86c3a45d6f508da7b1d660bb5901266b"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.060747 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "316e1b05a7a04d338b5f8b422927fbfd"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "1ec10cc52380257c4c017bfc318dc529"
server_key_iv: "86c3a45d6f508da7b1d660bb5901266b"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.064118 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 23:57:20.066192 18140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.067318 18117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 23:57:20.067402 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "316e1b05a7a04d338b5f8b422927fbfd"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "1ec10cc52380257c4c017bfc318dc529"
server_key_iv: "86c3a45d6f508da7b1d660bb5901266b"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.067473 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.090350 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.093374 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:20.093523 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.093991 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:20.094746 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:20.094806 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.094843 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:20.094859 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.103794 18117 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:33961
I20260806 23:57:20.103816 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:33961 every 8 connection(s)
I20260806 23:57:20.104676 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:20.113193 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 18117
I20260806 23:57:20.113291 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:20.113580 9009 external_mini_cluster.cc:1428] Setting key 34eb26ef09aa0f56662b51d61ba7ef03
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:20.118182 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.106462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:55013 (local address 127.8.204.126:38385)
0806 23:57:20.106604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.106608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.107416 (+ 808us) server_negotiation.cc:407] Connection header received
0806 23:57:20.108665 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.108669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.108729 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.108830 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.109936 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.110425 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.111668 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.111885 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.112008 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.114542 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.114552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.114554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.114583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.116155 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.116686 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.116690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.116692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.116772 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.117123 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.117126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.117128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.117299 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.117418 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.117892 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.118010 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:20.118922 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.106799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:55013)
0806 23:57:20.107271 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.107310 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.108465 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.108993 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.108999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.109329 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.109785 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.109797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.110549 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.110557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.111530 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.111542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.112061 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.112065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.112233 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.112768 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.112784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.114375 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.116293 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.116299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.116313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.116568 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.116866 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.116869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.116871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.117006 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.117420 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.117428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.117667 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.117846 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.118145 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260806 23:57:20.119918 18300 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
I20260806 23:57:20.120141 18300 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:20.120597 18300 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:20.121500 17660 ts_manager.cc:194] Registered new tserver with Master: 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:20.122027 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:55013
I20260806 23:57:20.127218 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.125.keytab.
Entry for principal oryx/127.8.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.125.keytab.
I20260806 23:57:20.153669 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:34409
--webserver_interface=127.8.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.125.keytab
--principal=oryx/127.8.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38385,127.8.204.125:34409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:20.259493 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:20.259742 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:20.259819 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:20.263654 18308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:20.263720 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:20.263754 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:20.263804 18308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:20.263839 18308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:20.268848 18308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38385,127.8.204.125:34409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.125.keytab
--principal=oryx/127.8.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
W20260806 23:57:20.269060 18308 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.204.126:38385,127.8.204.125:34409'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 23:57:20.269845 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:20.270833 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:20.278638 18321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.278666 18317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.278898 18308 server_base.cc:1034] running on GCE node
W20260806 23:57:20.279031 18316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.279585 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.280535 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.281729 18308 hybrid_clock.cc:648] HybridClock initialized: now 1786060640281701 us; error 50 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.284297 18308 init.cc:377] Logged in from keytab as oryx/127.8.204.125@KRBTEST.COM (short username oryx)
I20260806 23:57:20.285115 18308 webserver.cc:460] Webserver started at http://127.8.204.125:42181/ using document root <none> and password file <none>
I20260806 23:57:20.285624 18308 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.285722 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.285935 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:20.287752 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/instance:
uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "48ea5c950c87cc84755cdd2ccda868de"
server_key_iv: "504f7d21b17811f6f88a8683a0125467"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.288262 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal/instance:
uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "48ea5c950c87cc84755cdd2ccda868de"
server_key_iv: "504f7d21b17811f6f88a8683a0125467"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.291476 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:57:20.293710 18328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.294809 18308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 23:57:20.294931 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "48ea5c950c87cc84755cdd2ccda868de"
server_key_iv: "504f7d21b17811f6f88a8683a0125467"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.295027 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.324151 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.326592 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:20.326803 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.333258 18308 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.125:34409
I20260806 23:57:20.333529 18400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.125:34409 every 8 connection(s)
I20260806 23:57:20.334124 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
I20260806 23:57:20.336771 18401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:20.339437 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 18308
I20260806 23:57:20.340812 18401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.125@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
I20260806 23:57:20.351629 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.342472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.8.204.126:38385)
0806 23:57:20.342612 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.342617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.343018 (+ 401us) server_negotiation.cc:407] Connection header received
0806 23:57:20.343250 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.343253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.343298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.343364 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.344221 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.344736 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.345670 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.345888 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.345997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.348251 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.348254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.348256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.348280 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.349725 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.350184 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.350188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.350189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.350271 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.350620 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.350623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.350625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.350898 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.351003 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.351330 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.351442 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:57:20.352391 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.342414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58146)
0806 23:57:20.342885 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.342934 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.343124 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.343452 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.343459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.343886 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.344083 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.344095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.344865 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.344868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.345528 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.345537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.346031 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.346035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.346148 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.346668 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.346683 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.348068 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.349835 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.349846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.349855 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.350076 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.350362 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.350365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.350366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.350500 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.351002 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.351008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.351135 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.351287 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.351574 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260806 23:57:20.352396 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.340822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.8.204.125:34409)
0806 23:57:20.341252 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.341267 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.341299 (+ 32us) server_negotiation.cc:407] Connection header received
0806 23:57:20.341904 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.341921 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.342307 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.342696 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.343661 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.344584 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.345431 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.345780 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.345943 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.348561 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.348565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.348574 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.348594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.349932 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.350344 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.350348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.350349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.350453 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.350743 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.350746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.350749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.351051 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.351254 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.351420 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.351833 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260806 23:57:20.353672 18401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.125@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
I20260806 23:57:20.360810 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.354077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38768)
0806 23:57:20.354159 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.354171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.354238 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.354531 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.354535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.354665 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.354798 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.354808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.355809 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.355811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.356270 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.356277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.356663 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.356666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.356731 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.357151 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.357164 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.358270 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.359690 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.359692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.359694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.359874 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.360099 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.360101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.360102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.360171 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.360456 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.360458 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.360524 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.360607 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.360693 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 23:57:20.360844 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.354203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.8.204.125:34409)
0806 23:57:20.354330 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.354334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.354345 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:57:20.354377 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.354379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.354417 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.354487 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.354893 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.355682 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.356365 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.356568 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.356641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.358390 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.358393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.358394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.358415 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.359595 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.359954 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.359956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.359957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.360021 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.360245 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.360247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.360248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.360380 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.360450 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.360659 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.360740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:20.363407 18401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Bootstrap starting.
I20260806 23:57:20.365531 18401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:20.366271 18401 log.cc:824] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:20.367974 18401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: No bootstrap required, opened a new log
I20260806 23:57:20.371026 18401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } }
I20260806 23:57:20.371291 18401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:20.371352 18401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283f5e7abd3e4dcf8bbce2d66e9af981, State: Initialized, Role: FOLLOWER
I20260806 23:57:20.371742 18401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } }
I20260806 23:57:20.372200 18411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } } }
I20260806 23:57:20.372332 18411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:20.372848 18401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:57:20.377458 18424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:57:20.377542 18424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:57:20.379294 18308 master_runner.cc:428] Detected that this master 283f5e7abd3e4dcf8bbce2d66e9af981 is joining an existing cluster
I20260806 23:57:20.379366 18308 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.204.125:34409
I20260806 23:57:20.389165 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.381910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.8.204.125:34409)
0806 23:57:20.382024 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.382028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.382430 (+ 402us) server_negotiation.cc:407] Connection header received
0806 23:57:20.382571 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.382576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.382616 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.382679 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.383167 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.383606 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.384406 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.384608 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.384716 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.385903 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.385907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.385913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.385943 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.387507 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.387974 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.387978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.387979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.388061 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.388327 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.388330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.388332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.388480 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.388586 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.388874 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.388999 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":4992}
I20260806 23:57:20.389186 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.381598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38774)
0806 23:57:20.382351 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.382359 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.382474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.382781 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.382784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.382961 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.383054 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.383059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.383719 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.383723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.384289 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.384297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.384726 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.384729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.384826 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.385318 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.385335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.385764 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.387614 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.387619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.387621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.387868 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.388160 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.388163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.388165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.388256 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.388620 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.388623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.388702 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.388800 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.388948 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":25,"thread_start_us":95,"threads_started":1}
I20260806 23:57:20.389956 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.381591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.8.204.126:38385)
0806 23:57:20.381726 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.381730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.382181 (+ 451us) server_negotiation.cc:407] Connection header received
0806 23:57:20.382248 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.382251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.382302 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.382385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.383076 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.383857 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.385072 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.385385 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.385511 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.386557 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.386561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.386564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.386604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.388148 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.388559 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.388563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.388565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.388652 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.389014 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.389020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.389022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.389252 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.389320 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.389673 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.389803 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:57:20.390390 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.381548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58152)
0806 23:57:20.382091 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.382102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.382182 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.382528 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.382532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.382808 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.382942 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.382949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.383999 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.384002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.384897 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.384908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.385511 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.385515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.385605 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.386025 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.386040 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.386411 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.388236 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.388239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.388241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.388456 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.388749 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.388752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.388754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.388872 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.389394 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.389398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.389492 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.390121 (+ 629us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.390241 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.126@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
I20260806 23:57:20.402489 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.394197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38788)
0806 23:57:20.394500 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.394546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.394672 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.394980 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.394985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.395290 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.395391 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.395396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.396237 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.396240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.396773 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.396780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.397281 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.397284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.397397 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.397886 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.397897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.399325 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.401016 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.401025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.401029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.401247 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.401527 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.401529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.401530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.401643 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.402033 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.402038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.402132 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.402244 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.402376 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260806 23:57:20.402552 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.394229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.8.204.125:34409)
0806 23:57:20.394328 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.394332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.394624 (+ 292us) server_negotiation.cc:407] Connection header received
0806 23:57:20.394810 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.394813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.394861 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.394942 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.395494 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.396135 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.396888 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.397177 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.397267 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.399447 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.399451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.399453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.399478 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.400913 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.401350 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.401353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.401355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.401437 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.401736 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.401739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.401741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.401930 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.402027 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.402317 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.402419 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:20.403754 17660 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.204.125:34409: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
member_type: NON_VOTER
last_known_addr {
host: "127.8.204.125"
port: 34409
}
attrs {
promote: true
}
}
dest_uuid: "fd49e3b9534842259ff9f18c7fe9a82b"
cas_config_opid_index: -1
I20260806 23:57:20.404340 17660 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } }
I20260806 23:57:20.405875 18435 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } }
I20260806 23:57:20.408030 18435 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.204.125:34409
I20260806 23:57:20.408443 18308 master.cc:537] Master@127.8.204.125:34409 shutting down...
I20260806 23:57:20.408713 18366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:20.410140 18366 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 23:57:20.410473 18435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } }
I20260806 23:57:20.410616 18435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:20.410902 18438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:57:20.412863 17645 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 23:57:20.413848 17645 consensus_peers.cc:487] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b -> Peer 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125:34409): Couldn't send request to peer 283f5e7abd3e4dcf8bbce2d66e9af981. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 23:57:20.414188 18308 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:20.414458 18308 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:20.414552 18308 tablet_replica.cc:330] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: stopping tablet replica
WARNING: no policy specified for oryx/127.8.204.124@KRBTEST.COM; defaulting to no policy
I20260806 23:57:20.421303 18308 master.cc:552] Master@127.8.204.125:34409 shutdown complete.
I20260806 23:57:20.421350 18308 master_runner.cc:305] Clearing existing system tablet
I20260806 23:57:20.421578 18308 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.421667 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.124@KRBTEST.COM" created.
I20260806 23:57:20.423162 18308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:20.423923 18442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.424140 18308 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.424250 18308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:20.424278 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "48ea5c950c87cc84755cdd2ccda868de"
server_key_iv: "504f7d21b17811f6f88a8683a0125467"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.426028 18308 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:20.428088 18308 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 23:57:20.428159 18308 log.cc:1196] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 23:57:20.428395 18308 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Deleting consensus metadata
I20260806 23:57:20.428542 18308 master_runner.cc:315] Copying system tablet from 127.8.204.126:38385
I20260806 23:57:20.429603 18308 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: tablet copy: Beginning tablet copy session from remote peer at address 127.8.204.126:38385
I20260806 23:57:20.438313 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.430497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.8.204.126:38385)
0806 23:57:20.430604 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.430609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.431078 (+ 469us) server_negotiation.cc:407] Connection header received
0806 23:57:20.431257 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.431260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.431302 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.431348 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.431923 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.432570 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.433368 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.433577 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.433683 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.434756 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.434760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.434763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.434787 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.436287 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.436891 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.436895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.436897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.437021 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.437375 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.437378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.437380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.437592 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.437696 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.438027 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.438141 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:57:20.438452 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.430691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58166)
0806 23:57:20.431000 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.431014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.431146 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.431487 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.431490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.431711 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.431800 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.431806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.432692 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.432695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.433249 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.433255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.433686 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.433689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.433767 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.434265 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.434275 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.434554 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.436455 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.436458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.436461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.436755 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.437131 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.437134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.437136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.437286 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.437690 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.437692 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.437800 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.437950 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.438066 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":62,"threads_started":1}
I20260806 23:57:20.439056 17688 tablet_copy_service.cc:140] P fd49e3b9534842259ff9f18c7fe9a82b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 283f5e7abd3e4dcf8bbce2d66e9af981 ({username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:58166)
I20260806 23:57:20.439210 17688 tablet_copy_service.cc:161] P fd49e3b9534842259ff9f18c7fe9a82b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 283f5e7abd3e4dcf8bbce2d66e9af981 at {username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:58166: session id = 283f5e7abd3e4dcf8bbce2d66e9af981-00000000000000000000000000000000
I20260806 23:57:20.440490 17688 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 23:57:20.441952 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.124.keytab.
Entry for principal oryx/127.8.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.124.keytab.
I20260806 23:57:20.444033 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:39487
--webserver_interface=127.8.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.124.keytab
--principal=oryx/127.8.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:57:20.444412 18308 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: tablet copy: Starting download of 0 data blocks...
I20260806 23:57:20.444602 18308 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: tablet copy: Starting download of 1 WAL segments...
I20260806 23:57:20.447211 18308 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 23:57:20.448832 17688 tablet_copy_service.cc:342] P fd49e3b9534842259ff9f18c7fe9a82b: Request end of tablet copy session 283f5e7abd3e4dcf8bbce2d66e9af981-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:58166
I20260806 23:57:20.448940 17688 tablet_copy_service.cc:433] P fd49e3b9534842259ff9f18c7fe9a82b: ending tablet copy session 283f5e7abd3e4dcf8bbce2d66e9af981-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 283f5e7abd3e4dcf8bbce2d66e9af981
I20260806 23:57:20.456580 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:20.458045 18456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.458104 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.459781 18308 server_base.cc:1034] running on GCE node
W20260806 23:57:20.459905 18463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.460136 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.460410 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.461542 18308 hybrid_clock.cc:648] HybridClock initialized: now 1786060640461529 us; error 26 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.463132 18308 init.cc:377] Logged in from keytab as oryx/127.8.204.125@KRBTEST.COM (short username oryx)
I20260806 23:57:20.463562 18308 webserver.cc:460] Webserver started at http://127.8.204.125:45221/ using document root <none> and password file <none>
I20260806 23:57:20.463755 18308 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.463797 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.465111 18308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:20.466126 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.466353 18308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:20.466420 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "48ea5c950c87cc84755cdd2ccda868de"
server_key_iv: "504f7d21b17811f6f88a8683a0125467"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.466483 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.483623 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.485580 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.489915 18308 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.125:34409
I20260806 23:57:20.490082 18546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.125:34409 every 8 connection(s)
I20260806 23:57:20.490549 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
I20260806 23:57:20.492697 18550 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:57:20.493623 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Bootstrap starting.
I20260806 23:57:20.503089 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:57:20.503391 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Bootstrap complete.
I20260806 23:57:20.503962 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } }
I20260806 23:57:20.504122 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 283f5e7abd3e4dcf8bbce2d66e9af981, State: Initialized, Role: LEARNER
I20260806 23:57:20.504249 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } }
I20260806 23:57:20.504577 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } }
I20260806 23:57:20.504681 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:20.504760 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: LEARNER
I20260806 23:57:20.507745 18569 catalog_manager.cc:1259] Loaded cluster ID: ff2ee083c70e4140a725dd4276fbee83
I20260806 23:57:20.507848 18569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: loading cluster ID for follower catalog manager: success
I20260806 23:57:20.509888 18569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: acquiring CA information for follower catalog manager: success
I20260806 23:57:20.511226 18569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 23:57:20.564004 18455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:20.564271 18455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:20.564369 18455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:20.568408 18455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:20.568476 18455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:20.568512 18455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:20.568564 18455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:20.568600 18455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:20.573776 18455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.124.keytab
--principal=oryx/127.8.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:20.574687 18455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:20.575610 18455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:20.583925 18580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.583932 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.583945 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.584249 18455 server_base.cc:1034] running on GCE node
I20260806 23:57:20.584749 18455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.585707 18455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.586860 18455 hybrid_clock.cc:648] HybridClock initialized: now 1786060640586846 us; error 24 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.589473 18455 init.cc:377] Logged in from keytab as oryx/127.8.204.124@KRBTEST.COM (short username oryx)
I20260806 23:57:20.590318 18455 webserver.cc:460] Webserver started at http://127.8.204.124:34753/ using document root <none> and password file <none>
I20260806 23:57:20.590873 18455 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.590960 18455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.591255 18455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:20.593292 18455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/instance:
uuid: "124afff9a0894d868f28f657e0dc4b6c"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "0329f847f1e782f61f6862be951c2da2"
server_key_iv: "4f09347b80faa3f7635057a10fedf1a1"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.593832 18455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal/instance:
uuid: "124afff9a0894d868f28f657e0dc4b6c"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "0329f847f1e782f61f6862be951c2da2"
server_key_iv: "4f09347b80faa3f7635057a10fedf1a1"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.597126 18455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 23:57:20.599282 18588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.600349 18455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:20.600479 18455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "124afff9a0894d868f28f657e0dc4b6c"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "0329f847f1e782f61f6862be951c2da2"
server_key_iv: "4f09347b80faa3f7635057a10fedf1a1"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.600576 18455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.615885 18455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.618955 18455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:20.619150 18455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.626530 18455 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.124:39487
I20260806 23:57:20.626868 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.124:39487 every 8 connection(s)
I20260806 23:57:20.627563 18455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
I20260806 23:57:20.630259 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:20.631019 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 18455
I20260806 23:57:20.635020 18656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.124@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:20.646297 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.632680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.8.204.124:39487)
0806 23:57:20.633122 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.633139 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.633185 (+ 46us) server_negotiation.cc:407] Connection header received
0806 23:57:20.633823 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.633838 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.634149 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.634473 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.635301 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.636284 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.637358 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.637958 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.638130 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.640977 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.640983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.640993 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.641020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.643022 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.643580 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.643586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.643592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.643702 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.644003 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.644007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.644009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.644442 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.644722 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.645030 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.645336 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":141,"threads_started":1}
I20260806 23:57:20.646611 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.636576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58174)
0806 23:57:20.637002 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.637050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.637184 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.637566 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.637573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.638012 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.638148 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.638154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.639097 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.639100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.639825 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.639838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.640362 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.640366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.640507 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.641109 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.641122 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.642988 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.644896 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.644906 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.644912 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.645186 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.645490 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.645492 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.645493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.645570 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.646020 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.646026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.646148 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.646287 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.646442 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260806 23:57:20.646648 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.636738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.8.204.126:38385)
0806 23:57:20.636861 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.636867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.637150 (+ 283us) server_negotiation.cc:407] Connection header received
0806 23:57:20.637355 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.637358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.637412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.637494 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.638278 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.638959 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.639959 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.640179 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.640266 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.643143 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.643146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.643148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.643172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.644783 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.645294 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.645299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.645301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.645392 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.645669 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.645672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.645674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.645900 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.646002 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.646380 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.646495 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:20.648237 18656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.124@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
I20260806 23:57:20.656406 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.648744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38800)
0806 23:57:20.649282 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.649293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.649389 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.649645 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.649648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.649805 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.649889 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.649894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.650777 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.650783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.651282 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.651290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.651771 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.651775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.651870 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.652318 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.652338 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.653421 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.654984 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.654987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.654988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.655185 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.655479 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.655482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.655484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.655564 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.655934 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.655937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.656031 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.656143 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.656264 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474}
I20260806 23:57:20.656459 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.648895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.8.204.125:34409)
0806 23:57:20.649181 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.649185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.649353 (+ 168us) server_negotiation.cc:407] Connection header received
0806 23:57:20.649469 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.649472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.649522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.649622 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.650009 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.650644 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.651405 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.651655 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.651771 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.653559 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.653563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.653568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.653601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.654866 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.655285 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.655289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.655290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.655376 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.655659 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.655662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.655664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.655835 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.655945 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.656198 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.656313 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":136,"threads_started":1}
I20260806 23:57:20.657212 18656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.124@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
I20260806 23:57:20.664647 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.657584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:57038)
0806 23:57:20.657719 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.657733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.657809 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.658042 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.658044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.658198 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.658250 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.658255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.659438 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.659441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.659871 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.659878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.660287 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.660290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.660376 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.660746 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.660768 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.661809 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.663271 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.663278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.663280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.663463 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.663730 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.663733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.663735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.663818 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.664161 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.664164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.664248 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.664346 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.664475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260806 23:57:20.664647 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.657669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.8.204.124:39487)
0806 23:57:20.657777 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.657782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.657794 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:20.657912 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.657915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.657954 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.658030 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.658350 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.659320 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.659978 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.660187 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.660287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.661933 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.661936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.661939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.661966 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.663166 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.663556 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.663560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.663562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.663635 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.663905 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.663910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.663911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.664059 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.664169 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.664402 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.664525 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":41}
I20260806 23:57:20.667335 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Bootstrap starting.
I20260806 23:57:20.669528 18656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:20.670159 18656 log.cc:824] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:20.672297 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: No bootstrap required, opened a new log
I20260806 23:57:20.675901 18656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } }
I20260806 23:57:20.676087 18656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:20.676120 18656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124afff9a0894d868f28f657e0dc4b6c, State: Initialized, Role: FOLLOWER
I20260806 23:57:20.676505 18656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } }
I20260806 23:57:20.676911 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } } }
I20260806 23:57:20.677053 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:20.677544 18656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:57:20.681957 18677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:57:20.682044 18677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:57:20.685254 18455 master_runner.cc:428] Detected that this master 124afff9a0894d868f28f657e0dc4b6c is joining an existing cluster
I20260806 23:57:20.685312 18455 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.204.124:39487
I20260806 23:57:20.693384 17911 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
I20260806 23:57:20.700330 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17743
I20260806 23:57:20.714103 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:44059
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:38385
--webserver_port=36731
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:57:20.714154 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.687377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:57048)
0806 23:57:20.688410 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.688422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.688531 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.688757 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.688759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.688958 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.689040 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.689045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.689681 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.689685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.690207 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.690214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.690696 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.690699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.690825 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.691341 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.691355 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.692242 (+ 887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.712635 (+ 20393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.712640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.712643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.712902 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.713206 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.713209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.713210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.713280 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.713651 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.713654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.713725 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.713832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.713969 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":80,"threads_started":1}
I20260806 23:57:20.714561 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.8.204.124:39487)
0806 23:57:20.687864 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.687869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.688496 (+ 627us) server_negotiation.cc:407] Connection header received
0806 23:57:20.688611 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.688616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.688650 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.688744 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.689151 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.689582 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.690325 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.690583 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.690686 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.692382 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.692386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.692389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.692415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.712507 (+ 20092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.713021 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.713024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.713026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.713110 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.713357 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.713360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.713361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.713540 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.713620 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.714241 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.714358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":53}
I20260806 23:57:20.717656 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.687403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.8.204.126:38385)
0806 23:57:20.687522 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.687527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.687882 (+ 355us) server_negotiation.cc:407] Connection header received
0806 23:57:20.688061 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.688064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.688126 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.688180 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.688829 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.689614 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.690854 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.691223 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.691335 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.711499 (+ 20164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.711506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.711512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.711544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.713419 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.713932 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.713936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.713937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.714018 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.715535 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.715540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.715543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.715775 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.715849 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.717285 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.717448 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:20.718206 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.687331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58176)
0806 23:57:20.687803 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.687817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.687950 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.688350 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.688352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.688558 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.688699 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.688705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.689749 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.689752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.690662 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.690673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.691455 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.691460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.691569 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.692813 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.692824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.711314 (+ 18490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.713543 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.713549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.713554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.713804 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.714135 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.714138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.714139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.715065 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.716749 (+ 1684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.716752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.716953 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.717859 (+ 906us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.718001 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":37,"thread_start_us":116,"threads_started":1}
I20260806 23:57:20.720480 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.687955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.8.204.125:34409)
0806 23:57:20.688076 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.688080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.691017 (+ 2937us) server_negotiation.cc:407] Connection header received
0806 23:57:20.691212 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.691215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.691265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.691316 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.692033 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.692674 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.693911 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.694162 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.694272 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.714626 (+ 20354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.714630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.714632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.714658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.716555 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.718715 (+ 2160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.718830 (+ 115us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.718835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.718932 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.719321 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.719327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.719332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.719596 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.719686 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.720102 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.720255 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:20.721103 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.688115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38808)
0806 23:57:20.690946 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.690958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.691105 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.691455 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.691458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.691730 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.691852 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.691859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.692779 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.692782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.693719 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.693742 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.694285 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.694288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.694401 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.694844 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.694857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.714481 (+ 19624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.718300 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.718304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.718307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.718563 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.719074 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.719080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.719081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.719199 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.719772 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.719778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.719890 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.720693 (+ 803us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.720858 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2774,"thread_start_us":121,"threads_started":1}
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060639, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.126@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
I20260806 23:57:20.730948 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.721864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:57050)
0806 23:57:20.722027 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.722044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.722131 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.722482 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.722486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.722674 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.722806 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.722812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.723883 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.723886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.724733 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.724743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.725352 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.725356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.725446 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.725901 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.725917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.727525 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.729408 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.729412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.729413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.729655 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.729961 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.729964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.729966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.730054 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.730445 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.730448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.730539 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.730647 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.730799 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 23:57:20.730968 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.721956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.8.204.124:39487)
0806 23:57:20.722076 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.722081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.722096 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:20.722275 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.722280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.722349 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.722448 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.722945 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.723757 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.724859 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.725222 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.725339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.727690 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.727694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.727696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.727720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.729281 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.729778 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.729782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.729783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.729864 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.730157 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.730161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.730162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.730347 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.730442 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.730731 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.730836 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:57:20.732169 17660 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.204.124:39487: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c"
member_type: NON_VOTER
last_known_addr {
host: "127.8.204.124"
port: 39487
}
attrs {
promote: true
}
}
dest_uuid: "fd49e3b9534842259ff9f18c7fe9a82b"
cas_config_opid_index: 5
I20260806 23:57:20.732775 17660 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } }
I20260806 23:57:20.733779 18435 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.734338 18621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:20.735124 18435 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.204.124:39487
I20260806 23:57:20.735281 18435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.735363 18435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:20.735700 18621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 23:57:20.736423 18455 master.cc:537] Master@127.8.204.124:39487 shutting down...
I20260806 23:57:20.737046 18436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:57:20.738767 17640 consensus_peers.cc:487] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b -> Peer 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487): Couldn't send request to peer 124afff9a0894d868f28f657e0dc4b6c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 23:57:20.743294 18455 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:20.743552 18455 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:20.743798 18455 tablet_replica.cc:330] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: stopping tablet replica
I20260806 23:57:20.745961 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.734148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.8.204.125:34409)
0806 23:57:20.739330 (+ 5182us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.739336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.740384 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 23:57:20.740429 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.740431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.740491 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.740568 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:20.741965 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.742746 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.744228 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.744897 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.744977 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.745112 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.745786 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.745855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5102}
I20260806 23:57:20.746263 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.733929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:38816)
0806 23:57:20.740196 (+ 6267us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.740212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.740294 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.741715 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.741718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.741743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:20.741821 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.741827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.742901 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.742904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.744079 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.744091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.745210 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.745213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.745298 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.745490 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.746097 (+ 607us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.746168 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6187}
I20260806 23:57:20.746783 18513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 LEARNER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 23:57:20.747179 18436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 23:57:20.748970 18513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.750864 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.751047 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: LEARNER
I20260806 23:57:20.751010 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd49e3b9534842259ff9f18c7fe9a82b. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: NON_VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: true } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.751330 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: LEARNER
I20260806 23:57:20.751948 18436 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: attempting to promote NON_VOTER 283f5e7abd3e4dcf8bbce2d66e9af981 to VOTER
I20260806 23:57:20.752075 18455 master.cc:552] Master@127.8.204.124:39487 shutdown complete.
I20260806 23:57:20.752116 18455 master_runner.cc:305] Clearing existing system tablet
I20260806 23:57:20.752350 18455 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.752393 18455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.752354 18436 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } }
I20260806 23:57:20.753679 18512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 LEARNER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 23:57:20.753937 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.753172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:57066)
0806 23:57:20.753812 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.753878 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:39487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580}
I20260806 23:57:20.754074 18438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 23:57:20.754659 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.754339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:57070)
0806 23:57:20.754550 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.754600 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:39487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
W20260806 23:57:20.754813 17640 consensus_peers.cc:487] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b -> Peer 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487): Couldn't send request to peer 124afff9a0894d868f28f657e0dc4b6c. Status: Network error: Client connection negotiation failed: client connection to 127.8.204.124:39487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 23:57:20.755266 18455 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 23:57:20.756620 18435 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.757076 18512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.757959 18435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.758050 18435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:20.758157 18435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.758216 18435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:20.758464 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.758570 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:20.762868 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.763099 18455 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.763259 18455 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260806 23:57:20.763302 18455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "124afff9a0894d868f28f657e0dc4b6c"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "0329f847f1e782f61f6862be951c2da2"
server_key_iv: "4f09347b80faa3f7635057a10fedf1a1"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.765307 18455 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:20.767560 18455 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 23:57:20.767655 18455 log.cc:1196] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 23:57:20.767885 18455 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Deleting consensus metadata
I20260806 23:57:20.768036 18455 master_runner.cc:315] Copying system tablet from 127.8.204.126:38385
I20260806 23:57:20.769169 18455 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.204.126:38385
I20260806 23:57:20.778226 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.769722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.8.204.126:38385)
0806 23:57:20.769832 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.769836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.770071 (+ 235us) server_negotiation.cc:407] Connection header received
0806 23:57:20.770240 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.770243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.770292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.770374 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.770911 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.771813 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.773126 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.773459 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.773558 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.774757 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.774761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.774763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.774788 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.776302 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.776766 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.776770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.776772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.776854 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.777230 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.777235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.777239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.777447 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.777598 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.777879 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.777989 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:20.778362 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.769657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:58188)
0806 23:57:20.769981 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.769996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.770116 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.770412 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.770415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.770632 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.770790 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.770798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.771946 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.771951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.772980 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.772992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.773586 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.773590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.773679 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.774193 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.774206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.774590 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.776417 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.776420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.776423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.776666 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.777006 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.777009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.777011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.777105 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.777600 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.777603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.777701 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.777847 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.777984 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":80,"threads_started":1}
I20260806 23:57:20.779023 17688 tablet_copy_service.cc:140] P fd49e3b9534842259ff9f18c7fe9a82b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 124afff9a0894d868f28f657e0dc4b6c ({username='oryx', principal='oryx/127.8.204.124@KRBTEST.COM'} at 127.0.0.1:58188)
I20260806 23:57:20.779112 17688 tablet_copy_service.cc:161] P fd49e3b9534842259ff9f18c7fe9a82b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 124afff9a0894d868f28f657e0dc4b6c at {username='oryx', principal='oryx/127.8.204.124@KRBTEST.COM'} at 127.0.0.1:58188: session id = 124afff9a0894d868f28f657e0dc4b6c-00000000000000000000000000000000
I20260806 23:57:20.780161 17688 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 23:57:20.781422 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 23:57:20.783700 18455 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: tablet copy: Starting download of 0 data blocks...
I20260806 23:57:20.783881 18455 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: tablet copy: Starting download of 1 WAL segments...
I20260806 23:57:20.785379 18455 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 23:57:20.787096 17688 tablet_copy_service.cc:342] P fd49e3b9534842259ff9f18c7fe9a82b: Request end of tablet copy session 124afff9a0894d868f28f657e0dc4b6c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.204.124@KRBTEST.COM'} at 127.0.0.1:58188
I20260806 23:57:20.787174 17688 tablet_copy_service.cc:433] P fd49e3b9534842259ff9f18c7fe9a82b: ending tablet copy session 124afff9a0894d868f28f657e0dc4b6c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 124afff9a0894d868f28f657e0dc4b6c
I20260806 23:57:20.794028 18455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:20.795717 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.795791 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.796308 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.796257 18455 server_base.cc:1034] running on GCE node
I20260806 23:57:20.796625 18455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.796895 18455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.798046 18455 hybrid_clock.cc:648] HybridClock initialized: now 1786060640798031 us; error 29 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.799623 18455 init.cc:377] Logged in from keytab as oryx/127.8.204.124@KRBTEST.COM (short username oryx)
I20260806 23:57:20.800068 18455 webserver.cc:460] Webserver started at http://127.8.204.124:33477/ using document root <none> and password file <none>
I20260806 23:57:20.800253 18455 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.800299 18455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.801592 18455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 23:57:20.802621 18724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.802973 18455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:20.803053 18455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "124afff9a0894d868f28f657e0dc4b6c"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "0329f847f1e782f61f6862be951c2da2"
server_key_iv: "4f09347b80faa3f7635057a10fedf1a1"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.803131 18455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.816406 18455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.818616 18455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.824186 18455 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.124:39487
I20260806 23:57:20.824783 18455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
I20260806 23:57:20.827216 18804 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:57:20.828377 18804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Bootstrap starting.
I20260806 23:57:20.830945 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.124:39487 every 8 connection(s)
I20260806 23:57:20.839915 18804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:57:20.840266 18804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: Bootstrap complete.
I20260806 23:57:20.840804 18804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } }
I20260806 23:57:20.841009 18804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 124afff9a0894d868f28f657e0dc4b6c, State: Initialized, Role: LEARNER
I20260806 23:57:20.841172 18804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } }
I20260806 23:57:20.841773 18804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:20.841957 18810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:20.842123 18810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: LEARNER
I20260806 23:57:20.845115 18822 catalog_manager.cc:1259] Loaded cluster ID: ff2ee083c70e4140a725dd4276fbee83
I20260806 23:57:20.845165 18822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: loading cluster ID for follower catalog manager: success
I20260806 23:57:20.847215 18822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: acquiring CA information for follower catalog manager: success
I20260806 23:57:20.848500 18822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 23:57:20.857164 18692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:20.857353 18692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:20.857415 18692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:20.861208 18692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:20.861255 18692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:20.861340 18692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:20.866277 18692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:44059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=36731
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:20.867206 18692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:20.868134 18692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:20.877377 18692 server_base.cc:1034] running on GCE node
W20260806 23:57:20.877295 18834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.877306 18829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:20.877318 18830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:20.877969 18692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:20.878638 18692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:20.879860 18692 hybrid_clock.cc:648] HybridClock initialized: now 1786060640879847 us; error 23 us; skew 500 ppm
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:20.883199 18692 init.cc:377] Logged in from keytab as oryx/127.8.204.65@KRBTEST.COM (short username oryx)
I20260806 23:57:20.884172 18692 webserver.cc:460] Webserver started at http://127.8.204.65:36731/ using document root <none> and password file <none>
I20260806 23:57:20.884788 18692 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:20.884886 18692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:20.888429 18692 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:20.890254 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.891405 18692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:20.891494 18692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "eaa1cf5ed8cd4dc595518036ee5131ed"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "884a216e39e5b574264b997aa8b58c7f"
server_key_iv: "d6ed94b72da522071e6d36381153a703"
server_key_version: "encryptionkey@0"
I20260806 23:57:20.891963 18692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:20.901886 18101 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
I20260806 23:57:20.909271 18692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:20.911957 18692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:20.912178 18692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:20.912746 18692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:20.913612 18692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:20.913684 18692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.913757 18692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:20.913800 18692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:20.922343 18692 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:44059
I20260806 23:57:20.922394 19000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:44059 every 8 connection(s)
I20260806 23:57:20.923302 18692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:20.932677 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 18692
I20260806 23:57:20.932906 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17918
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
Aug 06 23:57:20 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060640, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:20.940403 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.924981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:55683 (local address 127.8.204.124:39487)
0806 23:57:20.925353 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.925359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.926552 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 23:57:20.927158 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.927161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.927214 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.927272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.928518 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929508 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930483 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930780 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.935451 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.935457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.935462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.935490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.937988 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.938637 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.938643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.938647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.938745 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.939102 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.939106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.939111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.939363 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.939435 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.940010 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.940128 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":79,"threads_started":1}
I20260806 23:57:20.941891 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.925302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.8.204.65:55683)
0806 23:57:20.926475 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.926484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.926953 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.927451 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.927458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.927873 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.928377 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.928384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929641 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.929644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.930397 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.930406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930989 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.931069 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.932009 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.932028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.935289 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.938161 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.938169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.938184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.938518 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.938848 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.938852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.938857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.939015 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.939512 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.939520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.939790 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.940597 (+ 807us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.940921 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":242,"spinlock_wait_cycles":21504,"thread_start_us":131,"threads_started":1}
I20260806 23:57:20.944039 19006 heartbeater.cc:347] Connected to a master server at 127.8.204.124:39487
I20260806 23:57:20.944367 19006 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:20.945084 19006 heartbeater.cc:511] Master 127.8.204.124:39487 requested a full tablet report, sending...
I20260806 23:57:20.945775 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.925301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:46223)
0806 23:57:20.925899 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.925947 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.926950 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.927451 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.927458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.927873 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.928375 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.928384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929444 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.929446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.930332 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.930342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930869 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.931057 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.931901 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.931912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.941407 (+ 9495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.943941 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.943945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.943947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.944210 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.944554 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.944559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.944560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.944712 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.945194 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.945200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.945316 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.945467 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.945595 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":170,"spinlock_wait_cycles":32256,"thread_start_us":129,"threads_started":1}
I20260806 23:57:20.946982 18751 ts_manager.cc:194] Registered new tserver with Master: eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059)
I20260806 23:57:20.947446 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.924847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46223 (local address 127.8.204.126:38385)
0806 23:57:20.925209 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.925214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.926036 (+ 822us) server_negotiation.cc:407] Connection header received
0806 23:57:20.927158 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.927161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.927219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.927270 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.928518 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929316 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930451 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930764 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930870 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.941568 (+ 10698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.941572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.941575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.941600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.943826 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.944331 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.944334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.944336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.944451 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.944810 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.944813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.944814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.945081 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.945226 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.945563 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.945731 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260806 23:57:20.947970 19005 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
I20260806 23:57:20.948055 19005 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:20.948223 19005 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:20.949050 17660 ts_manager.cc:194] Re-registered known tserver with Master: eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059)
I20260806 23:57:20.949827 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:46223
I20260806 23:57:20.952688 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.925526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:33587 (local address 127.8.204.125:34409)
0806 23:57:20.925670 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:20.925675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:20.926375 (+ 700us) server_negotiation.cc:407] Connection header received
0806 23:57:20.927145 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:20.927148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:20.927211 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:20.927286 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:20.928506 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929029 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930148 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930352 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.948868 (+ 18406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:20.948872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:20.948874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:20.948902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:20.950532 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.951181 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.951185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.951193 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.951284 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:20.951712 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:20.951716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:20.951718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:20.951936 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:20.952056 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:20.952480 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 23:57:20.952582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:57:20.952950 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:20.925302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.8.204.65:33587)
0806 23:57:20.926267 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:20.926278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:20.926951 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:20.927450 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:20.927456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:20.927870 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:20.928363 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.928377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.929191 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.929195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.930022 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:20.930030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:20.930860 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:20.930864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:20.931050 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:20.931632 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:20.931675 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:20.948707 (+ 17032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:20.950656 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.950662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.950667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.951043 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.951398 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:20.951404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:20.951409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:20.951540 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:20.952071 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:20.952078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:20.952245 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 23:57:20.952411 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 23:57:20.952640 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":62,"spinlock_wait_cycles":15744,"thread_start_us":155,"threads_started":1}
I20260806 23:57:20.953652 19007 heartbeater.cc:347] Connected to a master server at 127.8.204.125:34409
I20260806 23:57:20.953763 19007 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:20.953926 19007 heartbeater.cc:511] Master 127.8.204.125:34409 requested a full tablet report, sending...
I20260806 23:57:20.954954 18496 ts_manager.cc:194] Registered new tserver with Master: eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059)
I20260806 23:57:20.961978 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:44465
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:38385
--webserver_port=42661
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:21.071751 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:21.072031 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:21.072103 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:21.075878 19014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:21.075946 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:21.076041 19014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:21.081063 19014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=42661
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:21.082110 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:21.083151 19014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:21.092092 19027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:21.092160 19025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:21.092160 19024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:21.092412 19014 server_base.cc:1034] running on GCE node
I20260806 23:57:21.092811 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:21.093410 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:21.094563 19014 hybrid_clock.cc:648] HybridClock initialized: now 1786060641094548 us; error 23 us; skew 500 ppm
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:21.097138 19014 init.cc:377] Logged in from keytab as oryx/127.8.204.66@KRBTEST.COM (short username oryx)
I20260806 23:57:21.097944 19014 webserver.cc:460] Webserver started at http://127.8.204.66:42661/ using document root <none> and password file <none>
I20260806 23:57:21.098479 19014 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:21.098549 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:21.101926 19014 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:21.104125 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.105340 19014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:21.105450 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "f0d04faeb4524808804329390eab1ede"
format_stamp: "Formatted at 2026-08-06 23:57:19 on dist-test-slave-39l8"
server_key: "f672bab5f4db1646229122bff829450b"
server_key_iv: "67606faf3e23845eca5c9b398636ec1c"
server_key_version: "encryptionkey@0"
I20260806 23:57:21.106009 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:21.116171 19014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:21.118647 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:21.118891 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:21.119422 19014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:21.120167 19014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:21.120210 19014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.120271 19014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:21.120301 19014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.123807 18300 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
I20260806 23:57:21.128990 19014 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44465
I20260806 23:57:21.129035 19187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44465 every 8 connection(s)
I20260806 23:57:21.129870 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:21.138419 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 19014
I20260806 23:57:21.138624 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 18117
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
I20260806 23:57:21.150005 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.132324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:49453)
0806 23:57:21.133919 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.133928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.134045 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.134275 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.134278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.134682 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.135233 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.135247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.136045 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.136048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.136876 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.136885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.137337 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.137544 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.138142 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.138156 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.142863 (+ 4707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.145125 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.145130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.145140 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.147168 (+ 2028us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.147505 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.147507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.147509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.147687 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.148172 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.148179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.148448 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.148659 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.149000 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":49,"spinlock_wait_cycles":25984,"thread_start_us":85,"threads_started":1}
I20260806 23:57:21.150820 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.132423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49453 (local address 127.8.204.126:38385)
0806 23:57:21.132534 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.132538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.134023 (+ 1485us) server_negotiation.cc:407] Connection header received
0806 23:57:21.134136 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.134138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.134182 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.134260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.135376 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.135900 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.136990 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.137194 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137290 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.143069 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.143075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.143081 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.143115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.145012 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.147293 (+ 2281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.147299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.147303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.147390 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.147814 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.147819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.147823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.148058 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.148154 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.150500 (+ 2346us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.150627 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:21.151111 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:33961
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:38385
--webserver_port=34471
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:57:21.152729 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.131528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:54947 (local address 127.8.204.124:39487)
0806 23:57:21.131642 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.131647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.132939 (+ 1292us) server_negotiation.cc:407] Connection header received
0806 23:57:21.133819 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.133825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.133884 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.133983 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.135400 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.136151 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137475 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.137851 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.149109 (+ 11158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.149112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.149114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.149140 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.150820 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.151289 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.151293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.151296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.151379 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.151641 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.151644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.151646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.151899 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.152026 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.152357 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.152510 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:21.153223 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.132077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.8.204.66:54947)
0806 23:57:21.132786 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.132835 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.133645 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.134140 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.134149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.134748 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.135244 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.135253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.136290 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.136293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.137340 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.137350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.137963 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.138073 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.140982 (+ 2909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.140996 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.148943 (+ 7947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.150937 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.150939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.150941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.151148 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.151481 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.151483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.151484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.151547 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.152018 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.152024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.152136 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.152899 (+ 763us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.153034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":198,"spinlock_wait_cycles":13312,"thread_start_us":152,"threads_started":1}
I20260806 23:57:21.153663 19189 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
I20260806 23:57:21.153703 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.131528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48833 (local address 127.8.204.125:34409)
0806 23:57:21.131645 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.131649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.133174 (+ 1525us) server_negotiation.cc:407] Connection header received
0806 23:57:21.134111 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.134114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.134163 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.134253 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.135398 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.136149 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137467 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.137856 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.137965 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.148724 (+ 10759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.148730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.148735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.148765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.151732 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.152225 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.152229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.152231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.152318 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.152652 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.152658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.152661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.152945 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.153012 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.153398 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.153531 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:21.153918 19189 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.154212 19191 heartbeater.cc:347] Connected to a master server at 127.8.204.124:39487
I20260806 23:57:21.154318 19191 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.154352 19189 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:21.154534 19191 heartbeater.cc:511] Master 127.8.204.124:39487 requested a full tablet report, sending...
I20260806 23:57:21.155113 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.132077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.8.204.66:48833)
0806 23:57:21.133067 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.133077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.133632 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.134377 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.134380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.134686 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.135246 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.135253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.136290 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.136293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.137340 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.137350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.138122 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.138128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.138239 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.138600 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.138620 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.148579 (+ 9959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.151846 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.151849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.151851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.152124 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.152422 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.152424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.152426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.152511 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.153100 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.153104 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.153201 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.154796 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.154935 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":229,"spinlock_wait_cycles":10752,"thread_start_us":136,"threads_started":1}
I20260806 23:57:21.155441 17660 ts_manager.cc:194] Re-registered known tserver with Master: f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
I20260806 23:57:21.155741 19193 heartbeater.cc:347] Connected to a master server at 127.8.204.125:34409
I20260806 23:57:21.155730 18751 ts_manager.cc:194] Registered new tserver with Master: f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
I20260806 23:57:21.155807 19193 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.155936 19193 heartbeater.cc:511] Master 127.8.204.125:34409 requested a full tablet report, sending...
I20260806 23:57:21.156163 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:49453
I20260806 23:57:21.156553 18496 ts_manager.cc:194] Registered new tserver with Master: f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
W20260806 23:57:21.261286 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:21.261536 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:21.261606 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:21.265381 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:21.265452 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:21.265528 19201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:21.270577 19201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:33961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=34471
--tserver_master_addrs=127.8.204.126:38385,127.8.204.125:34409,127.8.204.124:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:21.271517 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:21.272462 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:21.281842 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:21.281879 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:21.281867 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:21.282006 19201 server_base.cc:1034] running on GCE node
I20260806 23:57:21.282495 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:21.283133 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:21.284312 19201 hybrid_clock.cc:648] HybridClock initialized: now 1786060641284295 us; error 31 us; skew 500 ppm
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:21.286844 19201 init.cc:377] Logged in from keytab as oryx/127.8.204.67@KRBTEST.COM (short username oryx)
I20260806 23:57:21.287556 19201 webserver.cc:460] Webserver started at http://127.8.204.67:34471/ using document root <none> and password file <none>
I20260806 23:57:21.288023 19201 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:21.288100 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:21.291416 19201 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:21.293236 19217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.294272 19201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:21.294390 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "316e1b05a7a04d338b5f8b422927fbfd"
format_stamp: "Formatted at 2026-08-06 23:57:20 on dist-test-slave-39l8"
server_key: "1ec10cc52380257c4c017bfc318dc529"
server_key_iv: "86c3a45d6f508da7b1d660bb5901266b"
server_key_version: "encryptionkey@0"
I20260806 23:57:21.294912 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:21.312026 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.308330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:33458)
0806 23:57:21.308554 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.308567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.308665 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.309031 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.309041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.309059 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.309148 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.309153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.310106 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.310108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.310884 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.310893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.311680 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.311683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.311776 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.311798 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.311840 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.311897 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260806 23:57:21.312054 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.308400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.8.204.124:39487)
0806 23:57:21.308533 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.308537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.308643 (+ 106us) server_negotiation.cc:407] Connection header received
0806 23:57:21.308810 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.308813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.308875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.308973 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.309263 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.310004 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.310990 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.311555 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.311648 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.311816 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.311862 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.311919 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:57:21.312609 18763 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 23:57:21.312994 18810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd49e3b9534842259ff9f18c7fe9a82b. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:21.313165 18810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: LEARNER
I20260806 23:57:21.319203 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:21.321933 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:21.322070 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:21.322547 19201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:21.323323 19201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:21.323366 19201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.323401 19201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:21.323419 19201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:21.331848 19201 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:33961
I20260806 23:57:21.331898 19380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:33961 every 8 connection(s)
I20260806 23:57:21.332778 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:21.339543 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 19201
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.125@KRBTEST.COM
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.124@KRBTEST.COM
Aug 06 23:57:21 dist-test-slave-39l8 krb5kdc[17593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060641, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:21.348842 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:56759 (local address 127.8.204.124:39487)
0806 23:57:21.334451 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.334455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.335934 (+ 1479us) server_negotiation.cc:407] Connection header received
0806 23:57:21.336577 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.336588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.336641 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.336733 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.338100 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338667 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.339773 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.340006 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.340092 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.343759 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.343763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.343767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.343795 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.346334 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.346970 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.346973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.346974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.347058 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.347438 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.347441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.347442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.347639 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.347734 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.348606 (+ 872us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.348704 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:57:21.350144 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.8.204.67:56759)
0806 23:57:21.335843 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.335854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.336399 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.336894 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.336899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.337393 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.337922 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.337957 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338843 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.338846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.339672 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.339685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.340233 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.340237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.340404 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.340913 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.340927 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.343580 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.346457 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.346462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.346474 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.346820 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.347179 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.347183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.347184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.347338 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.347788 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.347794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.348043 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.348227 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.348928 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":109,"spinlock_wait_cycles":26752,"thread_start_us":140,"threads_started":1}
I20260806 23:57:21.351387 19383 heartbeater.cc:347] Connected to a master server at 127.8.204.124:39487
I20260806 23:57:21.351609 19383 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.352165 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:43423 (local address 127.8.204.126:38385)
0806 23:57:21.334866 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.334870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.335970 (+ 1100us) server_negotiation.cc:407] Connection header received
0806 23:57:21.336603 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.336606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.336653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.336712 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.338076 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338829 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.342594 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.342962 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.343056 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.345888 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.345892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.345907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.345928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.347608 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.350756 (+ 3148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.350760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.350762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.350847 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.351175 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.351178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.351180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.351382 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.351453 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.351842 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.351969 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:21.352154 19383 heartbeater.cc:511] Master 127.8.204.124:39487 requested a full tablet report, sending...
I20260806 23:57:21.352739 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:43423)
0806 23:57:21.335883 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.335896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.336398 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.336895 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.336901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.337361 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.337938 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.337945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338952 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.338954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.342440 (+ 3486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.342451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.343255 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.343259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.343371 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.343995 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.344012 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.345731 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.348421 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.348424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.348426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.350612 (+ 2186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.350969 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.350972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.350974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.351064 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.351527 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.351530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.351618 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.352311 (+ 693us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.352458 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":253,"spinlock_wait_cycles":5760,"thread_start_us":109,"threads_started":1}
I20260806 23:57:21.353336 18751 ts_manager.cc:194] Registered new tserver with Master: 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:21.353360 19381 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38385
I20260806 23:57:21.353436 19381 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.353588 19381 heartbeater.cc:511] Master 127.8.204.126:38385 requested a full tablet report, sending...
I20260806 23:57:21.354178 17660 ts_manager.cc:194] Re-registered known tserver with Master: 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:21.354434 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:34893 (local address 127.8.204.125:34409)
0806 23:57:21.334456 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.334461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.335495 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 23:57:21.336631 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.336634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.336683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.336732 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.338076 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338805 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.339990 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.340280 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.340372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.344494 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.344499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.344504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.344534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.348295 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.349413 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.349416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.349417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.349507 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.349817 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.349822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.349823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.350021 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.350115 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.354202 (+ 4087us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.354309 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:21.354882 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.334861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.8.204.67:34893)
0806 23:57:21.335363 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.335408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.336397 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.336895 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.336901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.337361 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:21.337923 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.337938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.338945 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.338947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.339864 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.339878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.340481 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.340484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.340584 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.341074 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:21.341097 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:21.344331 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:21.349032 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.349035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.349037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.349284 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.349621 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:21.349625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:21.349626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:21.349708 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:21.350193 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:21.350198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:21.350306 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.354591 (+ 4285us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.354732 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":200,"spinlock_wait_cycles":4352,"thread_start_us":152,"threads_started":1}
I20260806 23:57:21.354869 17660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:43423
I20260806 23:57:21.355610 19384 heartbeater.cc:347] Connected to a master server at 127.8.204.125:34409
I20260806 23:57:21.355697 19384 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:21.356822 19384 heartbeater.cc:511] Master 127.8.204.125:34409 requested a full tablet report, sending...
I20260806 23:57:21.357249 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.342291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.8.204.125:34409)
0806 23:57:21.342631 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.342636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.343044 (+ 408us) server_negotiation.cc:407] Connection header received
0806 23:57:21.343372 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.343375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.343421 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.343481 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.345884 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.346620 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.347757 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.348051 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.348125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.349850 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.349856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.349861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.349883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.351612 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.355726 (+ 4114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.355729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.355732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.355809 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.356088 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.356091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.356092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.356282 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.356390 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.356723 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.356847 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":92,"threads_started":1}
I20260806 23:57:21.357385 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.341818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.8.204.126:38385)
0806 23:57:21.342907 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.342920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.343253 (+ 333us) server_negotiation.cc:407] Connection header received
0806 23:57:21.343287 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.343290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.343333 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.343415 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.345594 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.346396 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.347687 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.348026 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.348129 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.350049 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.350053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.350055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.350082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.354743 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.355514 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.355517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.355519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.355601 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.356266 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.356269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.356271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.356447 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.356571 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.357099 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.357218 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":66,"threads_started":1}
I20260806 23:57:21.357517 18496 ts_manager.cc:194] Registered new tserver with Master: 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:21.358922 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.342113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.8.204.124:39487)
0806 23:57:21.344261 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.344275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.344290 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:21.344328 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.344330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.344373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.351395 (+ 7022us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.351491 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.352224 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.353272 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.353557 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.353662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.355390 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.355393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.355395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.355417 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.356965 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.357439 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.357441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.357443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.357521 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.357859 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.357862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.357863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.358043 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.358181 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.358676 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.358824 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":75,"threads_started":1}
I20260806 23:57:21.360082 17660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:21.362799 17660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:21.365132 18438 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: NON_VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: true } } }
I20260806 23:57:21.365257 18438 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:21.368487 19406 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: attempting to promote NON_VOTER 124afff9a0894d868f28f657e0dc4b6c to VOTER
I20260806 23:57:21.368831 19406 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:21.370103 18763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 LEARNER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 23:57:21.370507 18512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Refusing update from remote peer fd49e3b9534842259ff9f18c7fe9a82b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 23:57:21.370610 19403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 23:57:21.371097 19403 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 23:57:21.373113 19405 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.373881 18512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.373769 18763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.374581 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.374670 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:21.374804 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.374871 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:21.374976 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.375128 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:21.376022 18810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.376144 18810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:21.376801 19416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:21.376924 19416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:21.383195 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.377869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44059 (local address 127.0.0.1:58440)
0806 23:57:21.378026 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.378040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.378120 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.378998 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.379001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.379030 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.379136 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.379143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.380505 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.380509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.381687 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.381716 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.382812 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.382816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.382914 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.382941 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.382990 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.383055 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":74}
I20260806 23:57:21.383955 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.378026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.8.204.65:44059)
0806 23:57:21.378470 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.378486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.378510 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:57:21.378590 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.378597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.378870 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.379027 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.379233 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.380374 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.381826 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.382653 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.383486 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.383639 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.383691 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.383785 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":125,"threads_started":1}
I20260806 23:57:21.384176 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.378948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.8.204.66:44465)
0806 23:57:21.380024 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.380040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.380058 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:57:21.380116 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.380120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.380242 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.380388 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.380564 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.381464 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.382609 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.383207 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.383347 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.383511 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.383927 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.384032 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":91,"threads_started":1}
I20260806 23:57:21.384363 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.378172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:33961 (local address 127.0.0.1:44852)
0806 23:57:21.378733 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.378747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.378854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.379547 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.379550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.379570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.379660 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.379666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.381296 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.381301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.382503 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.382515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.383621 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.383625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.383751 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.383803 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.383878 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.383957 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":52,"thread_start_us":94,"threads_started":1}
I20260806 23:57:21.384573 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.378579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.8.204.67:33961)
0806 23:57:21.379070 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.379081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.379121 (+ 40us) server_negotiation.cc:407] Connection header received
0806 23:57:21.379200 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.379206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.379376 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.379551 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.379800 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.380867 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.382665 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.383513 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.383631 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.383749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.384359 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.384449 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":118,"threads_started":1}
I20260806 23:57:21.384887 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.378492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44465 (local address 127.0.0.1:38488)
0806 23:57:21.379779 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.379792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.379883 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.380368 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.380371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.380389 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.380470 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.380477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.381589 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.381592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.382485 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.382494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.383589 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.383594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.383716 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.383800 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.383878 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.383957 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":90,"thread_start_us":85,"threads_started":1}
I20260806 23:57:21.386936 19287 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b (DEFAULT_TABLE table=test-table [id=23f7e0a50a30466599752e5c0fcb524a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:21.386942 18909 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b (DEFAULT_TABLE table=test-table [id=23f7e0a50a30466599752e5c0fcb524a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:21.387461 19099 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b (DEFAULT_TABLE table=test-table [id=23f7e0a50a30466599752e5c0fcb524a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:21.387925 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6566b251d94aefb8c3cd66d4c4904b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:21.387953 18909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6566b251d94aefb8c3cd66d4c4904b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:21.388368 19099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6566b251d94aefb8c3cd66d4c4904b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:21.394452 19426 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Bootstrap starting.
I20260806 23:57:21.395669 19427 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Bootstrap starting.
I20260806 23:57:21.396584 19428 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Bootstrap starting.
I20260806 23:57:21.396996 19426 tablet_bootstrap.cc:654] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:21.397861 19427 tablet_bootstrap.cc:654] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:21.397927 19426 log.cc:824] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:21.398588 19428 tablet_bootstrap.cc:654] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:21.398679 19427 log.cc:824] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:21.399453 19428 log.cc:824] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:21.400501 19426 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: No bootstrap required, opened a new log
I20260806 23:57:21.400636 19427 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: No bootstrap required, opened a new log
I20260806 23:57:21.400702 19426 ts_tablet_manager.cc:1397] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:57:21.400797 19427 ts_tablet_manager.cc:1397] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 23:57:21.401144 19428 tablet_bootstrap.cc:492] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: No bootstrap required, opened a new log
I20260806 23:57:21.401338 19428 ts_tablet_manager.cc:1397] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:57:21.404242 19428 raft_consensus.cc:348] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.404500 19428 raft_consensus.cc:374] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:21.404558 19428 raft_consensus.cc:729] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d04faeb4524808804329390eab1ede, State: Initialized, Role: FOLLOWER
I20260806 23:57:21.405014 19428 consensus_queue.cc:260] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.405277 19427 raft_consensus.cc:348] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.405303 19426 raft_consensus.cc:348] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.405509 19427 raft_consensus.cc:374] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:21.405553 19426 raft_consensus.cc:374] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:21.405562 19427 raft_consensus.cc:729] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa1cf5ed8cd4dc595518036ee5131ed, State: Initialized, Role: FOLLOWER
I20260806 23:57:21.405608 19426 raft_consensus.cc:729] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316e1b05a7a04d338b5f8b422927fbfd, State: Initialized, Role: FOLLOWER
I20260806 23:57:21.406026 19428 ts_tablet_manager.cc:1428] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:57:21.405958 19427 consensus_queue.cc:260] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.405994 19426 consensus_queue.cc:260] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.406378 19189 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
I20260806 23:57:21.406883 19427 ts_tablet_manager.cc:1428] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260806 23:57:21.406883 19426 ts_tablet_manager.cc:1428] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:57:21.407140 19381 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
I20260806 23:57:21.407140 19005 heartbeater.cc:503] Master 127.8.204.126:38385 was elected leader, sending a full tablet report...
W20260806 23:57:21.425053 19009 tablet.cc:2369] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:21.425711 19435 raft_consensus.cc:482] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:21.425859 19435 raft_consensus.cc:504] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.426820 19435 leader_election.cc:288] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059), f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
I20260806 23:57:21.430280 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.427061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44465 (local address 127.8.204.67:44041)
0806 23:57:21.427261 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.427275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.427351 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.427537 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.427540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.427553 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.427678 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.427682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.428348 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.428353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.429202 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.429211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.429850 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.429956 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.429981 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.430061 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.430136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":72}
I20260806 23:57:21.430292 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.427220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:44041 (local address 127.8.204.66:44465)
0806 23:57:21.427357 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.427362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.427381 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:57:21.427416 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.427417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.427446 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.427526 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.427787 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.428238 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429335 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.429728 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.429852 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.430109 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.430184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:57:21.430323 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.426975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44059 (local address 127.8.204.67:45451)
0806 23:57:21.427160 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:21.427173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:21.427279 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:21.427545 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:21.427548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:21.427562 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:21.427686 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.427691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.428376 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.428397 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.429215 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:21.429224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.429941 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:21.430028 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.430068 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 23:57:21.430146 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:57:21.430215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":40}
I20260806 23:57:21.430363 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.427084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:45451 (local address 127.8.204.65:44059)
0806 23:57:21.427346 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.427350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.427363 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:21.427405 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.427408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.427448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.427523 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:21.427817 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.428248 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429372 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.429815 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.429927 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.429972 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.430184 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.430249 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260806 23:57:21.430894 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6566b251d94aefb8c3cd66d4c4904b" candidate_uuid: "316e1b05a7a04d338b5f8b422927fbfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" is_pre_election: true
I20260806 23:57:21.430918 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6566b251d94aefb8c3cd66d4c4904b" candidate_uuid: "316e1b05a7a04d338b5f8b422927fbfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d04faeb4524808804329390eab1ede" is_pre_election: true
I20260806 23:57:21.431149 19123 raft_consensus.cc:2454] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316e1b05a7a04d338b5f8b422927fbfd in term 0.
I20260806 23:57:21.431143 18935 raft_consensus.cc:2454] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316e1b05a7a04d338b5f8b422927fbfd in term 0.
I20260806 23:57:21.431566 19220 leader_election.cc:302] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316e1b05a7a04d338b5f8b422927fbfd, eaa1cf5ed8cd4dc595518036ee5131ed; no voters:
I20260806 23:57:21.431838 19435 raft_consensus.cc:2781] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:21.431933 19435 raft_consensus.cc:482] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:21.432005 19435 raft_consensus.cc:3037] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:21.432965 19435 raft_consensus.cc:504] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.433326 19435 leader_election.cc:288] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [CANDIDATE]: Term 1 election: Requested vote from peers eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059), f0d04faeb4524808804329390eab1ede (127.8.204.66:44465)
I20260806 23:57:21.433646 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6566b251d94aefb8c3cd66d4c4904b" candidate_uuid: "316e1b05a7a04d338b5f8b422927fbfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed"
I20260806 23:57:21.433667 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6566b251d94aefb8c3cd66d4c4904b" candidate_uuid: "316e1b05a7a04d338b5f8b422927fbfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d04faeb4524808804329390eab1ede"
I20260806 23:57:21.433776 18935 raft_consensus.cc:3037] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:21.433782 19123 raft_consensus.cc:3037] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:21.434795 18935 raft_consensus.cc:2454] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316e1b05a7a04d338b5f8b422927fbfd in term 1.
I20260806 23:57:21.435110 19220 leader_election.cc:302] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316e1b05a7a04d338b5f8b422927fbfd, eaa1cf5ed8cd4dc595518036ee5131ed; no voters:
I20260806 23:57:21.435308 19435 raft_consensus.cc:2781] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:21.435421 19123 raft_consensus.cc:2454] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316e1b05a7a04d338b5f8b422927fbfd in term 1.
I20260806 23:57:21.435549 19435 raft_consensus.cc:686] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 1 LEADER]: Becoming Leader. State: Replica: 316e1b05a7a04d338b5f8b422927fbfd, State: Running, Role: LEADER
I20260806 23:57:21.435897 19435 consensus_queue.cc:237] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } }
I20260806 23:57:21.439352 17659 catalog_manager.cc:5495] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67). New cstate: current_term: 1 leader_uuid: "316e1b05a7a04d338b5f8b422927fbfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:21.460000 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.456982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.8.204.67:33961)
0806 23:57:21.457103 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.457106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.457119 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:21.457194 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.457196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.457242 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.457319 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:21.457532 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.458013 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.458837 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.459021 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.459093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.459329 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:21.459671 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:21.459752 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.459807 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.459848 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:57:21.465250 19123 raft_consensus.cc:1261] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Refusing update from remote peer 316e1b05a7a04d338b5f8b422927fbfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:21.465250 18935 raft_consensus.cc:1261] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Refusing update from remote peer 316e1b05a7a04d338b5f8b422927fbfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:21.465986 19435 consensus_queue.cc:1035] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:21.466212 19436 consensus_queue.cc:1035] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:21.475678 19443 mvcc.cc:205] Tried to move back new op lower bound from 7315704387435233280 to 7315704387324088320. Current Snapshot: MvccSnapshot[applied={T|T < 7315704387435233280}]
I20260806 23:57:21.478343 19440 mvcc.cc:205] Tried to move back new op lower bound from 7315704387435233280 to 7315704387324088320. Current Snapshot: MvccSnapshot[applied={T|T < 7315704387435233280 or (T in {7315704387435233280})}]
I20260806 23:57:21.484318 17659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:21.484444 17659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:21.488313 17659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd49e3b9534842259ff9f18c7fe9a82b: Sending DeleteTablet for 3 replicas of tablet 1b6566b251d94aefb8c3cd66d4c4904b
I20260806 23:57:21.489319 18909 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:21 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:58440
I20260806 23:57:21.489331 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:21 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44852
I20260806 23:57:21.489799 19452 tablet_replica.cc:330] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: stopping tablet replica
I20260806 23:57:21.490096 19099 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6566b251d94aefb8c3cd66d4c4904b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:21 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:38488
I20260806 23:57:21.490549 19454 tablet_replica.cc:330] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: stopping tablet replica
I20260806 23:57:21.490679 19452 raft_consensus.cc:2229] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:21.490904 19454 raft_consensus.cc:2229] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:21.491065 19452 raft_consensus.cc:2258] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:21.491521 19454 raft_consensus.cc:2258] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:21.492610 19453 tablet_replica.cc:330] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: stopping tablet replica
I20260806 23:57:21.492816 19452 ts_tablet_manager.cc:1905] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:21.492869 19453 raft_consensus.cc:2229] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:21.492904 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 17615
I20260806 23:57:21.500125 19452 ts_tablet_manager.cc:1918] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:21.500221 19452 log.cc:1196] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/1b6566b251d94aefb8c3cd66d4c4904b
I20260806 23:57:21.500537 19452 ts_tablet_manager.cc:1939] T 1b6566b251d94aefb8c3cd66d4c4904b P 316e1b05a7a04d338b5f8b422927fbfd: Deleting consensus metadata
I20260806 23:57:21.501686 19453 raft_consensus.cc:2258] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:21.505308 19453 ts_tablet_manager.cc:1905] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:21.507406 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.491886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.8.204.125:34409)
0806 23:57:21.493475 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.493481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.493498 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:57:21.493541 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.493547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.493612 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.493716 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.498625 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.499420 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.500648 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.500960 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.501041 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.503218 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.503223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.503226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.503251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.505256 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.505809 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.505812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.505814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.505900 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.506295 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.506298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.506300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.506578 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.506686 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.507126 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.507266 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499}
I20260806 23:57:21.508157 19453 ts_tablet_manager.cc:1918] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:21.508255 19453 log.cc:1196] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/1b6566b251d94aefb8c3cd66d4c4904b
I20260806 23:57:21.508611 19453 ts_tablet_manager.cc:1939] T 1b6566b251d94aefb8c3cd66d4c4904b P eaa1cf5ed8cd4dc595518036ee5131ed: Deleting consensus metadata
W20260806 23:57:21.508591 19219 connection.cc:535] server connection from 127.0.0.1:44852 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 23:57:21.509754 18845 connection.cc:441] server connection from 127.0.0.1:58440 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58440 (request call id 1) could send its response
I20260806 23:57:21.511340 19454 ts_tablet_manager.cc:1905] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:21.513943 19454 ts_tablet_manager.cc:1918] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:21.514091 19454 log.cc:1196] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/1b6566b251d94aefb8c3cd66d4c4904b
I20260806 23:57:21.514443 19454 ts_tablet_manager.cc:1939] T 1b6566b251d94aefb8c3cd66d4c4904b P f0d04faeb4524808804329390eab1ede: Deleting consensus metadata
W20260806 23:57:21.515446 19041 connection.cc:441] server connection from 127.0.0.1:38488 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38488 (request call id 1) could send its response
I20260806 23:57:21.516618 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.491467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.8.204.124:39487)
0806 23:57:21.493355 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.493363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.507936 (+ 14573us) server_negotiation.cc:407] Connection header received
0806 23:57:21.508161 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.508165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.508228 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.508287 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.509265 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.510016 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.511026 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.511253 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.511360 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.513167 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.513170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.513172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.513197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.514932 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.515401 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.515407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.515411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.515493 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.515782 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.515786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.515788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.515987 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.516112 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.516352 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.516478 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1789}
I20260806 23:57:21.522691 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.513914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.8.204.124:39487)
0806 23:57:21.514042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.514047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.515258 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 23:57:21.515429 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.515431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.515472 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.515519 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.516277 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.516905 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.517741 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.517949 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.518051 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.519478 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.519482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.519484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.519506 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.521026 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.521465 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.521468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.521470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.521541 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.521858 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.521861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.521862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.522040 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.522134 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.522444 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.522546 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:21.529322 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:21.520696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.8.204.125:34409)
0806 23:57:21.520792 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:21.520796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:21.521396 (+ 600us) server_negotiation.cc:407] Connection header received
0806 23:57:21.521851 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:21.521855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:21.521897 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:21.521943 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:21.522891 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.523625 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.524835 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:21.525031 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:21.525115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:21.526479 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:21.526482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:21.526484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:21.526504 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:21.527816 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.528259 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.528262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.528263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.528331 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:21.528595 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:21.528597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:21.528598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:21.528738 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:21.528826 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:21.529069 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 23:57:21.529155 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:57:22.477525 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.476927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:57869)
0806 23:57:22.477299 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.477410 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
W20260806 23:57:22.477834 19189 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:38385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.478255 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.477825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:59837)
0806 23:57:22.478075 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.478174 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":59,"threads_started":1}
W20260806 23:57:22.478597 19381 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:38385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.478940 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.478450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:45619)
0806 23:57:22.478752 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.478837 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":52,"threads_started":1}
W20260806 23:57:22.479192 19005 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:38385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.989331 19473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd49e3b9534842259ff9f18c7fe9a82b)
I20260806 23:57:22.989482 19474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fd49e3b9534842259ff9f18c7fe9a82b)
I20260806 23:57:22.989586 19473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:22.989684 19474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:22.990561 19473 leader_election.cc:288] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487)
I20260806 23:57:22.990585 19474 leader_election.cc:288] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125:34409)
I20260806 23:57:22.991189 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.990734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41672)
0806 23:57:22.991004 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.991082 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":56,"threads_started":1}
W20260806 23:57:22.991366 18482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.991648 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.990905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41686)
0806 23:57:22.991489 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.991559 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":52,"threads_started":1}
W20260806 23:57:22.991802 18728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.992350 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.992091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41688)
0806 23:57:22.992234 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.992283 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 23:57:22.992512 18482 leader_election.cc:334] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.992690 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.992468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41692)
0806 23:57:22.992587 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.992628 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 23:57:22.992832 18728 leader_election.cc:334] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.995015 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.991077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.8.204.124:39487)
0806 23:57:22.991344 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:22.991349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:22.991363 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:22.991411 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:22.991413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:22.991458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:22.991534 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:22.991964 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.992754 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.993987 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.994499 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.994613 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:22.994671 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:22.994882 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 23:57:22.994939 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"spinlock_wait_cycles":2432,"thread_start_us":60,"threads_started":1}
I20260806 23:57:22.995041 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.990758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:39487 (local address 127.0.0.1:33494)
0806 23:57:22.991217 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.991231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:22.991331 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:22.991670 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:22.991673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:22.991706 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:22.991806 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:22.991815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.992855 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.992859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:22.993875 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:22.993884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.994636 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.994639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:22.994736 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:22.994790 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 23:57:22.994865 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 23:57:22.994926 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":20,"thread_start_us":72,"threads_started":1}
I20260806 23:57:22.995643 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124afff9a0894d868f28f657e0dc4b6c" is_pre_election: true
I20260806 23:57:22.995823 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.990878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34409 (local address 127.0.0.1:41332)
0806 23:57:22.991404 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.991418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:22.991557 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:22.992213 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:22.992216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:22.992236 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:22.992333 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:22.992339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.993587 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.993589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:22.994514 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:22.994523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.995416 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.995420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:22.995504 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:22.995539 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 23:57:22.995634 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 23:57:22.995707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":62,"threads_started":1}
I20260806 23:57:22.995823 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.991131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.8.204.125:34409)
0806 23:57:22.991881 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:22.991886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:22.991947 (+ 61us) server_negotiation.cc:407] Connection header received
0806 23:57:22.992017 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:22.992023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:22.992072 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:22.992537 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:22.992723 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.993481 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.994625 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:22.995254 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:22.995388 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:22.995441 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:22.995651 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 23:57:22.995719 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":46,"threads_started":1}
I20260806 23:57:22.995877 18763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283f5e7abd3e4dcf8bbce2d66e9af981 in term 1.
I20260806 23:57:22.996253 18478 leader_election.cc:302] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124afff9a0894d868f28f657e0dc4b6c, 283f5e7abd3e4dcf8bbce2d66e9af981; no voters: fd49e3b9534842259ff9f18c7fe9a82b
I20260806 23:57:22.996277 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124afff9a0894d868f28f657e0dc4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" is_pre_election: true
I20260806 23:57:22.996479 18512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124afff9a0894d868f28f657e0dc4b6c in term 1.
I20260806 23:57:22.996670 19473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 23:57:22.996747 19473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fd49e3b9534842259ff9f18c7fe9a82b)
I20260806 23:57:22.996788 19473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:57:22.996882 18729 leader_election.cc:302] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124afff9a0894d868f28f657e0dc4b6c, 283f5e7abd3e4dcf8bbce2d66e9af981; no voters: fd49e3b9534842259ff9f18c7fe9a82b
I20260806 23:57:22.997085 19474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 23:57:22.997129 19474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fd49e3b9534842259ff9f18c7fe9a82b)
I20260806 23:57:22.997154 19474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:57:22.997953 19473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:22.998035 19474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:22.998329 19473 leader_election.cc:288] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 election: Requested vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487)
I20260806 23:57:22.998401 19474 leader_election.cc:288] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 election: Requested vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 283f5e7abd3e4dcf8bbce2d66e9af981 (127.8.204.125:34409)
I20260806 23:57:22.998627 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.998427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41708)
0806 23:57:22.998500 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.998567 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 23:57:22.998941 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124afff9a0894d868f28f657e0dc4b6c"
I20260806 23:57:22.999121 18763 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 283f5e7abd3e4dcf8bbce2d66e9af981 in current term 2: Already voted for candidate 124afff9a0894d868f28f657e0dc4b6c in this term.
I20260806 23:57:22.999183 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124afff9a0894d868f28f657e0dc4b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981"
I20260806 23:57:22.999281 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.999032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41738)
0806 23:57:22.999172 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.999220 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 23:57:22.999337 18512 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 124afff9a0894d868f28f657e0dc4b6c in current term 2: Already voted for candidate 283f5e7abd3e4dcf8bbce2d66e9af981 in this term.
W20260806 23:57:22.999395 18482 leader_election.cc:334] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:22.999502 18478 leader_election.cc:302] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 283f5e7abd3e4dcf8bbce2d66e9af981; no voters: 124afff9a0894d868f28f657e0dc4b6c, fd49e3b9534842259ff9f18c7fe9a82b
I20260806 23:57:22.999639 19473 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 23:57:22.999768 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:22.998665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41724)
0806 23:57:22.998805 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:22.998853 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 23:57:23.000336 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:23.000133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41740)
0806 23:57:23.000239 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:23.000283 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 23:57:23.000437 18728 leader_election.cc:334] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:23.000499 18728 leader_election.cc:302] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 124afff9a0894d868f28f657e0dc4b6c; no voters: 283f5e7abd3e4dcf8bbce2d66e9af981, fd49e3b9534842259ff9f18c7fe9a82b
I20260806 23:57:23.000604 19474 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 23:57:23.478972 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:23.478403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:35963)
0806 23:57:23.478782 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:23.478855 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260806 23:57:23.479866 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:23.479354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:35907)
0806 23:57:23.479696 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:23.479762 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":81,"threads_started":1}
I20260806 23:57:23.480206 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:23.479742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:48661)
0806 23:57:23.480031 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:23.480095 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":72,"threads_started":1}
I20260806 23:57:24.480213 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.479797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:34417)
0806 23:57:24.480030 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.480100 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260806 23:57:24.481598 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.480524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:57979)
0806 23:57:24.481441 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.481505 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":64,"threads_started":1}
I20260806 23:57:24.481751 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.481045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:58467)
0806 23:57:24.481368 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.481426 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":60,"threads_started":1}
I20260806 23:57:24.847529 19494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:24.847653 19494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:24.848111 19494 leader_election.cc:288] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487)
I20260806 23:57:24.848668 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.848260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41970)
0806 23:57:24.848509 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.848575 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260806 23:57:24.848608 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124afff9a0894d868f28f657e0dc4b6c" is_pre_election: true
I20260806 23:57:24.848764 18763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283f5e7abd3e4dcf8bbce2d66e9af981 in term 2.
I20260806 23:57:24.849120 18478 leader_election.cc:302] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124afff9a0894d868f28f657e0dc4b6c, 283f5e7abd3e4dcf8bbce2d66e9af981; no voters:
I20260806 23:57:24.849327 19494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 23:57:24.849407 19494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:24.849458 19494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:57:24.849581 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.849341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41980)
0806 23:57:24.849452 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.849513 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 23:57:24.849725 18482 leader_election.cc:334] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:24.850546 19494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:24.850895 19494 leader_election.cc:288] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 election: Requested vote from peers fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385), 124afff9a0894d868f28f657e0dc4b6c (127.8.204.124:39487)
I20260806 23:57:24.851197 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.850989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:41996)
0806 23:57:24.851076 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.851138 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 23:57:24.851208 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124afff9a0894d868f28f657e0dc4b6c"
I20260806 23:57:24.851315 18763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:57:24.851872 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.851676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:42010)
0806 23:57:24.851778 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.851821 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 23:57:24.851979 18482 leader_election.cc:334] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
I20260806 23:57:24.852402 18763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283f5e7abd3e4dcf8bbce2d66e9af981 in term 3.
I20260806 23:57:24.852691 18478 leader_election.cc:302] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124afff9a0894d868f28f657e0dc4b6c, 283f5e7abd3e4dcf8bbce2d66e9af981; no voters: fd49e3b9534842259ff9f18c7fe9a82b
I20260806 23:57:24.852874 19494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 23:57:24.853034 19494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [term 3 LEADER]: Becoming Leader. State: Replica: 283f5e7abd3e4dcf8bbce2d66e9af981, State: Running, Role: LEADER
I20260806 23:57:24.853307 19494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } }
I20260806 23:57:24.854321 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283f5e7abd3e4dcf8bbce2d66e9af981. Latest consensus state: current_term: 3 leader_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:24.854451 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:24.854847 19501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:24.856964 19501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:24.857360 19501 catalog_manager.cc:1259] Loaded cluster ID: ff2ee083c70e4140a725dd4276fbee83
I20260806 23:57:24.857425 19501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:24.857959 19501 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:24.858346 19501 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Loaded TSK: 0
I20260806 23:57:24.859045 19501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:24.885787 18496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:24.888881 18496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:24.891870 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.891608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:42036)
0806 23:57:24.891741 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.891797 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 23:57:24.891894 18763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [term 3 FOLLOWER]: Refusing update from remote peer 283f5e7abd3e4dcf8bbce2d66e9af981: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 23:57:24.892418 19497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 23:57:24.892616 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.892366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.0.0.1:42050)
0806 23:57:24.892478 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.892540 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 23:57:24.892738 18482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 -> Peer fd49e3b9534842259ff9f18c7fe9a82b (127.8.204.126:38385): Couldn't send request to peer fd49e3b9534842259ff9f18c7fe9a82b. Status: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 23:57:24.893952 19505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283f5e7abd3e4dcf8bbce2d66e9af981. Latest consensus state: current_term: 3 leader_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:24.894058 19505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:24.895172 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:24.895349 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:24.895507 19505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fd49e3b9534842259ff9f18c7fe9a82b" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38385 } } peers { permanent_uuid: "283f5e7abd3e4dcf8bbce2d66e9af981" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34409 } attrs { promote: false } } peers { permanent_uuid: "124afff9a0894d868f28f657e0dc4b6c" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 39487 } attrs { promote: false } } }
I20260806 23:57:24.895750 19505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124afff9a0894d868f28f657e0dc4b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:57:24.905603 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.900715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44465 (local address 127.0.0.1:38502)
0806 23:57:24.900836 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.900849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:24.900944 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:24.901920 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:24.901922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:24.901936 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:24.902032 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.902041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.903223 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.903227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.904239 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.904252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.905191 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.905276 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.905301 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:57:24.905353 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 23:57:24.905433 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 23:57:24.905906 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.901003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:33961 (local address 127.0.0.1:44862)
0806 23:57:24.901215 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.901230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:24.901318 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:24.901819 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:24.901823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:24.901846 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:24.901957 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.901963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.903069 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.903073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.904143 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.904154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.905131 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.905228 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.905257 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 23:57:24.905328 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 23:57:24.905402 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260806 23:57:24.905920 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.901265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.8.204.66:44465)
0806 23:57:24.901678 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.901682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.901695 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:24.901743 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.901746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.901791 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.902145 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:24.902252 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.903090 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.904381 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.905056 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905598 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.905663 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.905714 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.905785 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":79,"threads_started":1}
I20260806 23:57:24.906242 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.901128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.8.204.67:33961)
0806 23:57:24.901361 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.901365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.901380 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:24.901550 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.901555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.901608 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.901685 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:24.902120 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.902940 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.904301 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.904908 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905007 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.905074 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.905542 (+ 468us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.905614 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260806 23:57:24.906466 19099 tablet_service.cc:1467] Processing CreateTablet for tablet e6c6b92142804e6bba429ac7dd4d774a (DEFAULT_TABLE table=test-table [id=468cd5773a184262b80897f8b94f343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:24.906528 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.901416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44059 (local address 127.0.0.1:58444)
0806 23:57:24.902193 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.902208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:24.902297 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:24.902733 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:24.902736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:24.902753 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:24.902838 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.902844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.903860 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.903864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.904922 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.904935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.905947 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.906225 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.906243 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:57:24.906328 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 23:57:24.906390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":83,"threads_started":1}
I20260806 23:57:24.906538 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.901670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.8.204.65:44059)
0806 23:57:24.901925 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.901931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.902400 (+ 469us) server_negotiation.cc:407] Connection header received
0806 23:57:24.902464 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.902467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.902522 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.902720 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:24.902959 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.903749 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905067 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.905746 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.905834 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.905887 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.906357 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.906429 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260806 23:57:24.906798 19099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c6b92142804e6bba429ac7dd4d774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:24.906829 19287 tablet_service.cc:1467] Processing CreateTablet for tablet e6c6b92142804e6bba429ac7dd4d774a (DEFAULT_TABLE table=test-table [id=468cd5773a184262b80897f8b94f343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:24.907040 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c6b92142804e6bba429ac7dd4d774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:24.909035 19520 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Bootstrap starting.
I20260806 23:57:24.909251 19521 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Bootstrap starting.
I20260806 23:57:24.909842 18909 tablet_service.cc:1467] Processing CreateTablet for tablet e6c6b92142804e6bba429ac7dd4d774a (DEFAULT_TABLE table=test-table [id=468cd5773a184262b80897f8b94f343c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:24.910125 18909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c6b92142804e6bba429ac7dd4d774a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:24.910135 19520 tablet_bootstrap.cc:654] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:24.910125 19521 tablet_bootstrap.cc:654] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:24.911437 19521 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: No bootstrap required, opened a new log
I20260806 23:57:24.911530 19521 ts_tablet_manager.cc:1397] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:24.911717 19520 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: No bootstrap required, opened a new log
I20260806 23:57:24.911808 19520 ts_tablet_manager.cc:1397] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:24.911966 19521 raft_consensus.cc:348] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.912114 19521 raft_consensus.cc:374] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:24.912158 19521 raft_consensus.cc:729] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316e1b05a7a04d338b5f8b422927fbfd, State: Initialized, Role: FOLLOWER
I20260806 23:57:24.912323 19521 consensus_queue.cc:260] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.912339 19520 raft_consensus.cc:348] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.912436 19520 raft_consensus.cc:374] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:24.912467 19520 raft_consensus.cc:729] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d04faeb4524808804329390eab1ede, State: Initialized, Role: FOLLOWER
I20260806 23:57:24.912715 19521 ts_tablet_manager.cc:1428] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 23:57:24.912635 19520 consensus_queue.cc:260] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.913318 19520 ts_tablet_manager.cc:1428] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:24.913507 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.913153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:32983)
0806 23:57:24.913385 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.913445 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260806 23:57:24.914021 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.913737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:50959)
0806 23:57:24.913902 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.913955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 23:57:24.915072 19527 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed: Bootstrap starting.
I20260806 23:57:24.915751 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.914736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:51969)
0806 23:57:24.915626 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.915677 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":818}
I20260806 23:57:24.916082 19527 tablet_bootstrap.cc:654] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:24.916414 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.916157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:35633)
0806 23:57:24.916294 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.916351 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 23:57:24.916581 19384 heartbeater.cc:503] Master 127.8.204.125:34409 was elected leader, sending a full tablet report...
I20260806 23:57:24.917248 19193 heartbeater.cc:503] Master 127.8.204.125:34409 was elected leader, sending a full tablet report...
I20260806 23:57:24.917738 19527 tablet_bootstrap.cc:492] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed: No bootstrap required, opened a new log
I20260806 23:57:24.917848 19527 ts_tablet_manager.cc:1397] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:24.918234 19527 raft_consensus.cc:348] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.918355 19527 raft_consensus.cc:374] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:24.918391 19527 raft_consensus.cc:729] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa1cf5ed8cd4dc595518036ee5131ed, State: Initialized, Role: FOLLOWER
I20260806 23:57:24.918538 19527 consensus_queue.cc:260] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.918947 19527 ts_tablet_manager.cc:1428] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:57:24.919499 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.919179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:49993)
0806 23:57:24.919378 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.919431 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 23:57:24.920195 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.919955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:51045)
0806 23:57:24.920089 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.920135 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 23:57:24.920197 19007 heartbeater.cc:503] Master 127.8.204.125:34409 was elected leader, sending a full tablet report...
I20260806 23:57:24.929613 19533 raft_consensus.cc:482] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:24.929697 19533 raft_consensus.cc:504] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.930907 19533 leader_election.cc:288] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d04faeb4524808804329390eab1ede (127.8.204.66:44465), 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:24.935353 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.931235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:44501 (local address 127.8.204.67:33961)
0806 23:57:24.931338 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.931341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.931593 (+ 252us) server_negotiation.cc:407] Connection header received
0806 23:57:24.931711 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.931717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.931763 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.931846 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:24.932121 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.932777 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.934122 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.934704 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.934829 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.934872 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.935150 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.935215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:24.935352 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.931127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:52151 (local address 127.8.204.66:44465)
0806 23:57:24.931247 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.931253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.931269 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:57:24.931312 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.931315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.931360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.931431 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:24.931734 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.932519 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.933991 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.934597 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.934725 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.934798 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.935150 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.935215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:57:24.935370 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.931006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:44465 (local address 127.8.204.65:52151)
0806 23:57:24.931085 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.931101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:24.931190 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:24.931467 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:24.931470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:24.931490 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:24.931629 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.931638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.932625 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.932627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.933873 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.933883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.934784 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.934789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.934944 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.935044 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 23:57:24.935120 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 23:57:24.935211 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":90}
I20260806 23:57:24.935381 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.931127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:33961 (local address 127.8.204.65:44501)
0806 23:57:24.931480 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.931491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:24.931599 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:24.931870 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:24.931872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:24.931887 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:24.931960 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.931964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.932872 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.932875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.933984 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:24.933992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.934805 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.934807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:24.934940 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.935042 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 23:57:24.935120 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:57:24.935212 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":63,"thread_start_us":96,"threads_started":1}
I20260806 23:57:24.935792 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c6b92142804e6bba429ac7dd4d774a" candidate_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d04faeb4524808804329390eab1ede" is_pre_election: true
I20260806 23:57:24.935906 19123 raft_consensus.cc:2454] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa1cf5ed8cd4dc595518036ee5131ed in term 0.
I20260806 23:57:24.936079 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c6b92142804e6bba429ac7dd4d774a" candidate_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316e1b05a7a04d338b5f8b422927fbfd" is_pre_election: true
I20260806 23:57:24.936311 18843 leader_election.cc:302] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaa1cf5ed8cd4dc595518036ee5131ed, f0d04faeb4524808804329390eab1ede; no voters:
I20260806 23:57:24.936323 19315 raft_consensus.cc:2454] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa1cf5ed8cd4dc595518036ee5131ed in term 0.
I20260806 23:57:24.936532 19533 raft_consensus.cc:2781] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:24.936606 19533 raft_consensus.cc:482] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:24.936671 19533 raft_consensus.cc:3037] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:24.937553 19533 raft_consensus.cc:504] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.937865 19533 leader_election.cc:288] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [CANDIDATE]: Term 1 election: Requested vote from peers f0d04faeb4524808804329390eab1ede (127.8.204.66:44465), 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961)
I20260806 23:57:24.938149 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c6b92142804e6bba429ac7dd4d774a" candidate_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d04faeb4524808804329390eab1ede"
I20260806 23:57:24.938277 19123 raft_consensus.cc:3037] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:24.938359 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c6b92142804e6bba429ac7dd4d774a" candidate_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316e1b05a7a04d338b5f8b422927fbfd"
I20260806 23:57:24.938472 19315 raft_consensus.cc:3037] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:24.939324 19123 raft_consensus.cc:2454] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa1cf5ed8cd4dc595518036ee5131ed in term 1.
I20260806 23:57:24.939530 19315 raft_consensus.cc:2454] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa1cf5ed8cd4dc595518036ee5131ed in term 1.
I20260806 23:57:24.939621 18843 leader_election.cc:302] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaa1cf5ed8cd4dc595518036ee5131ed, f0d04faeb4524808804329390eab1ede; no voters:
I20260806 23:57:24.939788 19533 raft_consensus.cc:2781] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:24.939985 19533 raft_consensus.cc:686] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [term 1 LEADER]: Becoming Leader. State: Replica: eaa1cf5ed8cd4dc595518036ee5131ed, State: Running, Role: LEADER
I20260806 23:57:24.940222 19533 consensus_queue.cc:237] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } }
I20260806 23:57:24.940591 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.940366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:53623)
0806 23:57:24.940490 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.940535 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 23:57:24.942785 18496 catalog_manager.cc:5495] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed reported cstate change: term changed from 0 to 1, leader changed from <none> to eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65). New cstate: current_term: 1 leader_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaa1cf5ed8cd4dc595518036ee5131ed" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 44059 } health_report { overall_health: HEALTHY } } }
I20260806 23:57:24.955860 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.952844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.8.204.65:44059)
0806 23:57:24.952960 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:24.952963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:24.952975 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:24.953111 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:24.953113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:24.953160 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:24.953258 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:24.953457 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.953884 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.954651 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:24.954904 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:24.954980 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:24.955201 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:24.955580 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:24.955662 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:24.955723 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 23:57:24.955766 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:24.958263 19123 raft_consensus.cc:1261] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Refusing update from remote peer eaa1cf5ed8cd4dc595518036ee5131ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:24.958525 19315 raft_consensus.cc:1261] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Refusing update from remote peer eaa1cf5ed8cd4dc595518036ee5131ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:24.958822 19535 consensus_queue.cc:1035] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d04faeb4524808804329390eab1ede" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:24.959146 19533 consensus_queue.cc:1035] T e6c6b92142804e6bba429ac7dd4d774a P eaa1cf5ed8cd4dc595518036ee5131ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "316e1b05a7a04d338b5f8b422927fbfd" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:24.960649 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.960324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:41673)
0806 23:57:24.960501 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.960570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 23:57:24.962841 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.962527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:47915)
0806 23:57:24.962678 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.962773 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 23:57:24.963441 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.963147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:39957)
0806 23:57:24.963332 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.963381 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 23:57:24.964088 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.963794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.66:56881)
0806 23:57:24.963969 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.964022 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 23:57:24.966368 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.966110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.65:51729)
0806 23:57:24.966255 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.966306 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 23:57:24.968134 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:24.967905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38385 (local address 127.8.204.67:52863)
0806 23:57:24.968007 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.968063 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 23:57:24.972702 19543 mvcc.cc:205] Tried to move back new op lower bound from 7315704401744449536 to 7315704401677729792. Current Snapshot: MvccSnapshot[applied={T|T < 7315704401744449536 or (T in {7315704401744449536})}]
I20260806 23:57:24.973402 18496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:24.973522 18496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:24.977162 18496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 283f5e7abd3e4dcf8bbce2d66e9af981: Sending DeleteTablet for 3 replicas of tablet e6c6b92142804e6bba429ac7dd4d774a
I20260806 23:57:24.977828 19099 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c6b92142804e6bba429ac7dd4d774a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:24 UTC) from {username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:38502
I20260806 23:57:24.978056 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c6b92142804e6bba429ac7dd4d774a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:24 UTC) from {username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:44862
I20260806 23:57:24.978461 18909 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c6b92142804e6bba429ac7dd4d774a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:24 UTC) from {username='oryx', principal='oryx/127.8.204.125@KRBTEST.COM'} at 127.0.0.1:58444
I20260806 23:57:24.978658 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 18692
I20260806 23:57:24.986261 19551 tablet_replica.cc:330] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: stopping tablet replica
I20260806 23:57:24.986442 19551 raft_consensus.cc:2229] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:24.986618 19551 raft_consensus.cc:2258] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:24.986934 19553 tablet_replica.cc:330] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: stopping tablet replica
I20260806 23:57:24.987078 19553 raft_consensus.cc:2229] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:24.987208 19553 raft_consensus.cc:2258] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:24.988108 19551 ts_tablet_manager.cc:1905] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:24.988713 19553 ts_tablet_manager.cc:1905] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:24.991410 19551 ts_tablet_manager.cc:1918] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:24.991497 19551 log.cc:1196] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/e6c6b92142804e6bba429ac7dd4d774a
I20260806 23:57:24.991772 19551 ts_tablet_manager.cc:1939] T e6c6b92142804e6bba429ac7dd4d774a P f0d04faeb4524808804329390eab1ede: Deleting consensus metadata
I20260806 23:57:24.992141 19553 ts_tablet_manager.cc:1918] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:24.992287 19553 log.cc:1196] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/e6c6b92142804e6bba429ac7dd4d774a
I20260806 23:57:24.992671 19553 ts_tablet_manager.cc:1939] T e6c6b92142804e6bba429ac7dd4d774a P 316e1b05a7a04d338b5f8b422927fbfd: Deleting consensus metadata
I20260806 23:57:24.992666 18478 catalog_manager.cc:4841] TS f0d04faeb4524808804329390eab1ede (127.8.204.66:44465): tablet e6c6b92142804e6bba429ac7dd4d774a (table test-table [id=468cd5773a184262b80897f8b94f343c]) successfully deleted
I20260806 23:57:24.993454 18480 catalog_manager.cc:4841] TS 316e1b05a7a04d338b5f8b422927fbfd (127.8.204.67:33961): tablet e6c6b92142804e6bba429ac7dd4d774a (table test-table [id=468cd5773a184262b80897f8b94f343c]) successfully deleted
W20260806 23:57:24.997560 19517 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:24.997248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:44059 (local address 127.0.0.1:58452)
0806 23:57:24.997378 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:24.997439 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:44059: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 23:57:24.997614 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 19014
W20260806 23:57:25.014971 18482 catalog_manager.cc:4584] TS eaa1cf5ed8cd4dc595518036ee5131ed (127.8.204.65:44059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6c6b92142804e6bba429ac7dd4d774a: Network error: Client connection negotiation failed: client connection to 127.8.204.65:44059: connect: Connection reset by peer (error 104)
I20260806 23:57:25.024276 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 19201
I20260806 23:57:25.036020 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 18308
I20260806 23:57:25.048732 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 18455
2026-08-06T23:57:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7062 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:25 dist-test-slave-39l8 krb5kdc[19557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:25 dist-test-slave-39l8 krb5kdc[19557](info): set up 2 sockets
Aug 06 23:57:25 dist-test-slave-39l8 krb5kdc[19557](info): commencing operation
krb5kdc: starting...
W20260806 23:57:26.203877 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab.
Entry for principal oryx/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab.
I20260806 23:57:26.316926 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40461
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab
--principal=oryx/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:26.424468 19574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:26.424700 19574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:26.424736 19574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:26.428403 19574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:26.428442 19574 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:57:26.428457 19574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:26.428472 19574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:26.428486 19574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:26.428499 19574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:26.432963 19574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40461
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.126.keytab
--principal=oryx/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:26.433804 19574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:26.434680 19574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:26.442571 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:26.442579 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:26.442843 19574 server_base.cc:1034] running on GCE node
W20260806 23:57:26.442935 19583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:26.443431 19574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:26.444440 19574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:26.445623 19574 hybrid_clock.cc:648] HybridClock initialized: now 1786060646445602 us; error 39 us; skew 500 ppm
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:26.452669 19574 init.cc:377] Logged in from keytab as oryx/127.8.204.126@KRBTEST.COM (short username oryx)
I20260806 23:57:26.453490 19574 webserver.cc:460] Webserver started at http://127.8.204.126:37795/ using document root <none> and password file <none>
I20260806 23:57:26.453972 19574 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:26.454030 19574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:26.454185 19574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:26.456048 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "964f224a465249c7b34422b0e2c2d812"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "7c429ea977e2bf468605b221d37136b6"
server_key_iv: "5de96ef5aa67629387ddc3d704123549"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.456550 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "964f224a465249c7b34422b0e2c2d812"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "7c429ea977e2bf468605b221d37136b6"
server_key_iv: "5de96ef5aa67629387ddc3d704123549"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.459800 19574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:26.462007 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:26.463231 19574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:26.463356 19574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "964f224a465249c7b34422b0e2c2d812"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "7c429ea977e2bf468605b221d37136b6"
server_key_iv: "5de96ef5aa67629387ddc3d704123549"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.463461 19574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:26.488277 19574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:26.491295 19574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:26.491493 19574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:26.499217 19574 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40461
I20260806 23:57:26.499233 19685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40461 every 8 connection(s)
I20260806 23:57:26.500130 19574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:26.502777 19686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:26.503451 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 19574
I20260806 23:57:26.503592 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:26.503907 9009 external_mini_cluster.cc:1428] Setting key 5668b4835dc8956cac2f980bf95b1c9c
I20260806 23:57:26.509251 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Bootstrap starting.
I20260806 23:57:26.511637 19686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:26.512355 19686 log.cc:824] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:26.514526 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: No bootstrap required, opened a new log
I20260806 23:57:26.517277 19686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } }
I20260806 23:57:26.517525 19686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:26.517633 19686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964f224a465249c7b34422b0e2c2d812, State: Initialized, Role: FOLLOWER
I20260806 23:57:26.518074 19686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } }
I20260806 23:57:26.518218 19686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:26.518309 19686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:26.518421 19686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:26.518806 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.505273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.8.204.126:40461)
0806 23:57:26.505666 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:26.505683 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:26.505723 (+ 40us) server_negotiation.cc:407] Connection header received
0806 23:57:26.506527 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:26.506550 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:26.506957 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:26.507303 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:26.507933 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.509366 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.510147 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.510785 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.510969 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.513558 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:26.513564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:26.513578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:26.513606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:26.515922 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.516379 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.516384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.516388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.516511 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.516806 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.516809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.516811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.517175 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:26.517386 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:26.517689 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 23:57:26.517968 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260806 23:57:26.519316 19686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } }
I20260806 23:57:26.519629 19686 leader_election.cc:302] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964f224a465249c7b34422b0e2c2d812; no voters:
I20260806 23:57:26.519936 19686 leader_election.cc:288] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:26.520035 19695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:26.520253 19695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [term 1 LEADER]: Becoming Leader. State: Replica: 964f224a465249c7b34422b0e2c2d812, State: Running, Role: LEADER
I20260806 23:57:26.520609 19695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } }
I20260806 23:57:26.521059 19686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:26.521727 19696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964f224a465249c7b34422b0e2c2d812. Latest consensus state: current_term: 1 leader_uuid: "964f224a465249c7b34422b0e2c2d812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } } }
I20260806 23:57:26.521847 19696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:26.522181 19699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964f224a465249c7b34422b0e2c2d812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964f224a465249c7b34422b0e2c2d812" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40461 } } }
I20260806 23:57:26.522306 19699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:26.522625 19708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:26.525702 19708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:26.530859 19708 catalog_manager.cc:1347] Generated new cluster ID: 1a51b14113534f0b9f2414bda6ede64a
I20260806 23:57:26.530920 19708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:26.542488 19708 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:26.543453 19708 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:26.548658 19708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Generated new TSK 0
I20260806 23:57:26.549281 19708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
I20260806 23:57:26.576871 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40461
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:26.685344 19727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:26.685568 19727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:26.685649 19727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:26.689422 19727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:26.689468 19727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:26.689538 19727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:57:26.689611 19727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:26.694399 19727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:40461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:26.695287 19727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:26.696177 19727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:26.705030 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:26.705049 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:26.705096 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:26.705180 19727 server_base.cc:1034] running on GCE node
I20260806 23:57:26.705724 19727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:26.706279 19727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:26.707458 19727 hybrid_clock.cc:648] HybridClock initialized: now 1786060646707432 us; error 36 us; skew 500 ppm
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:26.709865 19727 init.cc:377] Logged in from keytab as oryx/127.8.204.65@KRBTEST.COM (short username oryx)
I20260806 23:57:26.710611 19727 webserver.cc:460] Webserver started at http://127.8.204.65:43135/ using document root <none> and password file <none>
I20260806 23:57:26.711215 19727 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:26.711304 19727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:26.711519 19727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:26.713426 19727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "f3bbff8ace5944f3841e2579e2210145"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "186a6173ec1be411bd7b30b4facaf2fb"
server_key_iv: "327d80c1d67c04d76f49ef3d9edf7cb5"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.713964 19727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "f3bbff8ace5944f3841e2579e2210145"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "186a6173ec1be411bd7b30b4facaf2fb"
server_key_iv: "327d80c1d67c04d76f49ef3d9edf7cb5"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.717255 19727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 23:57:26.719432 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:26.720528 19727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:26.720661 19727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "f3bbff8ace5944f3841e2579e2210145"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "186a6173ec1be411bd7b30b4facaf2fb"
server_key_iv: "327d80c1d67c04d76f49ef3d9edf7cb5"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.720759 19727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:26.735124 19727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:26.737500 19727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:26.737676 19727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:26.738948 19727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:26.739027 19727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:26.739114 19727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:26.739156 19727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:26.749218 19727 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:45587
I20260806 23:57:26.749253 19919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:45587 every 8 connection(s)
I20260806 23:57:26.750170 19727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:26.752312 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.739649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:44517 (local address 127.8.204.126:40461)
0806 23:57:26.739793 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:26.739798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:26.740577 (+ 779us) server_negotiation.cc:407] Connection header received
0806 23:57:26.741391 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:26.741396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:26.741451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:26.741547 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:26.743125 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.743895 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.745524 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.745920 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.746029 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.748774 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:26.748777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:26.748780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:26.748805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:26.750312 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.750862 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.750867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.750870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.750957 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.751274 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.751277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.751279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.751444 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:26.751543 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:26.751963 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 23:57:26.752125 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:57:26.753028 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.740003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.65:44517)
0806 23:57:26.740426 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:26.740470 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:26.741194 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:26.741709 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:26.741718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:26.742164 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:26.742936 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.742959 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.744029 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.744034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.745389 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.745403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.746110 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.746116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.746321 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.746864 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:26.746885 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:26.748616 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:26.750405 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.750410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.750422 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.750749 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.751040 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.751042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.751044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.751169 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.751543 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:26.751550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:26.751757 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 23:57:26.751910 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 23:57:26.752254 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
I20260806 23:57:26.753086 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 19727
I20260806 23:57:26.753208 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:26.753540 9009 external_mini_cluster.cc:1428] Setting key 32404b59c631ce3b97511a9ed0e0d8d1
I20260806 23:57:26.754448 19921 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40461
I20260806 23:57:26.754570 19921 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:26.755061 19921 heartbeater.cc:511] Master 127.8.204.126:40461 requested a full tablet report, sending...
I20260806 23:57:26.756587 19622 ts_manager.cc:194] Registered new tserver with Master: f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587)
I20260806 23:57:26.757694 19622 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:44517
WARNING: no policy specified for oryx/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:57:26.766116 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.756428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:51991 (local address 127.8.204.126:40461)
0806 23:57:26.758111 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:26.758113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:26.758123 (+ 10us) server_negotiation.cc:407] Connection header received
0806 23:57:26.758172 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:26.758175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:26.758212 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:26.758290 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:26.758770 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.759188 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.761552 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.761952 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.762074 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.763131 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:26.763137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:26.763142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:26.763171 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:26.764676 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.765086 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.765089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.765090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.765157 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.765391 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.765393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.765394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.765530 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:26.765659 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:26.765871 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 23:57:26.765952 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1636}
I20260806 23:57:26.766166 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.756390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.65:51991)
0806 23:57:26.756697 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:26.756712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:26.756781 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:26.758332 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:26.758335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:26.758522 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:26.758622 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.758629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.760959 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.760961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.761438 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.761444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.762085 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.762088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.762158 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.762641 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:26.762653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:26.762990 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:26.764797 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.764801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.764805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.764984 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.765242 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.765244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.765245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.765316 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.765607 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:26.765609 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:26.765713 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 23:57:26.765819 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 23:57:26.765936 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":170,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
I20260806 23:57:26.781672 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40461
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:26.886912 19932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:26.887123 19932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:26.887164 19932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:26.890799 19932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:26.890837 19932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:26.890870 19932 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:57:26.890897 19932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:26.895493 19932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:40461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:26.896253 19932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:26.897089 19932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:26.905869 19932 server_base.cc:1034] running on GCE node
W20260806 23:57:26.905803 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:26.905840 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:26.905802 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:26.906461 19932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:26.907085 19932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:26.908318 19932 hybrid_clock.cc:648] HybridClock initialized: now 1786060646908304 us; error 31 us; skew 500 ppm
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:26.911037 19932 init.cc:377] Logged in from keytab as oryx/127.8.204.66@KRBTEST.COM (short username oryx)
I20260806 23:57:26.911793 19932 webserver.cc:460] Webserver started at http://127.8.204.66:42131/ using document root <none> and password file <none>
I20260806 23:57:26.912345 19932 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:26.912405 19932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:26.912647 19932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:26.914492 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "3d392a0e48b24b09ac002babde098971"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "2ae097e941295f210eb6140393f79b81"
server_key_iv: "d5529b1112a7b6cb2175fd1ad550f37f"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.915079 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "3d392a0e48b24b09ac002babde098971"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "2ae097e941295f210eb6140393f79b81"
server_key_iv: "d5529b1112a7b6cb2175fd1ad550f37f"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.918416 19932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:26.920625 19954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:26.921880 19932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:26.921990 19932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "3d392a0e48b24b09ac002babde098971"
format_stamp: "Formatted at 2026-08-06 23:57:26 on dist-test-slave-39l8"
server_key: "2ae097e941295f210eb6140393f79b81"
server_key_iv: "d5529b1112a7b6cb2175fd1ad550f37f"
server_key_version: "encryptionkey@0"
I20260806 23:57:26.922080 19932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:26.941322 19932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:26.943977 19932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:26.944159 19932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:26.945482 19932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:26.945529 19932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:26.945602 19932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:26.945636 19932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:57:26 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:26.956633 19932 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:32911
I20260806 23:57:26.956708 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:32911 every 8 connection(s)
I20260806 23:57:26.957556 19932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:26.958745 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.946377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:51601 (local address 127.8.204.126:40461)
0806 23:57:26.946497 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:26.946501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:26.947437 (+ 936us) server_negotiation.cc:407] Connection header received
0806 23:57:26.948507 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:26.948510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:26.948570 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:26.948649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:26.949980 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.950822 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.951931 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.952371 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.952498 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.955181 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:26.955185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:26.955187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:26.955217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:26.956792 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.957233 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.957238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.957243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.957329 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.957709 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.957713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.957725 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.957886 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:26.957986 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:26.958487 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 23:57:26.958570 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:26.959383 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.946863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.66:51601)
0806 23:57:26.947286 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:26.947323 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:26.948285 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:26.948821 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:26.948831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:26.949274 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:26.949813 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.949833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.950957 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.950963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.951801 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.951810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.952568 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.952574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.952778 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.953350 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:26.953376 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:26.955020 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:26.956898 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.956902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.956911 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.957126 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.957418 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.957420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.957421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.957622 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.957986 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:26.957993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:26.958268 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 23:57:26.958437 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 23:57:26.958675 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260806 23:57:26.960444 20134 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40461
I20260806 23:57:26.960584 20134 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:26.961000 20134 heartbeater.cc:511] Master 127.8.204.126:40461 requested a full tablet report, sending...
I20260806 23:57:26.962052 19623 ts_manager.cc:194] Registered new tserver with Master: 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911)
I20260806 23:57:26.962872 19623 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:51601
I20260806 23:57:26.967893 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 19932
I20260806 23:57:26.968031 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:26.968398 9009 external_mini_cluster.cc:1428] Setting key 00cabdc36b03750b249c3e29b9ddb1ab
I20260806 23:57:26.969053 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.962501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:59011 (local address 127.8.204.126:40461)
0806 23:57:26.962604 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:26.962608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:26.962819 (+ 211us) server_negotiation.cc:407] Connection header received
0806 23:57:26.962916 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:26.962918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:26.962951 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:26.963037 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:26.963514 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.963992 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.964785 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.964983 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.965082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.966019 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:26.966023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:26.966025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:26.966047 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:26.967358 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.967789 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.967792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.967794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.967880 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:26.968220 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:26.968224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:26.968226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:26.968496 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:26.968596 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:26.968822 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 23:57:26.968913 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:57:26.969071 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:26.962425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.66:59011)
0806 23:57:26.962734 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:26.962748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:26.962838 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:26.963151 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:26.963153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:26.963309 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:26.963399 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.963404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.964130 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.964133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.964670 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:26.964677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:26.965104 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:26.965107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:26.965197 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:26.965619 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:26.965631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:26.965889 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:26.967467 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.967472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.967474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.967676 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.968001 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:26.968004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:26.968006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:26.968120 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:26.968582 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:26.968585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:26.968655 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 23:57:26.968738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:57:26.968875 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
I20260806 23:57:26.993465 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40461
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:27.099469 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:27.099675 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:27.099720 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:27.103734 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:27.103806 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:27.103844 20149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:57:27.103875 20149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:27.108753 20149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.126:40461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:27.109596 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:27.110466 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:27.119316 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:27.119304 20159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:27.119505 20149 server_base.cc:1034] running on GCE node
W20260806 23:57:27.119304 20160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:27.120051 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:27.120690 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:27.121870 20149 hybrid_clock.cc:648] HybridClock initialized: now 1786060647121854 us; error 30 us; skew 500 ppm
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:27.124567 20149 init.cc:377] Logged in from keytab as oryx/127.8.204.67@KRBTEST.COM (short username oryx)
I20260806 23:57:27.125418 20149 webserver.cc:460] Webserver started at http://127.8.204.67:38003/ using document root <none> and password file <none>
I20260806 23:57:27.126030 20149 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:27.126116 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:27.126338 20149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:27.128249 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
format_stamp: "Formatted at 2026-08-06 23:57:27 on dist-test-slave-39l8"
server_key: "e79b9c5adf15ed697ad6dad50d048d10"
server_key_iv: "9d9708fac90b8765127c65187f690b82"
server_key_version: "encryptionkey@0"
I20260806 23:57:27.128778 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
format_stamp: "Formatted at 2026-08-06 23:57:27 on dist-test-slave-39l8"
server_key: "e79b9c5adf15ed697ad6dad50d048d10"
server_key_iv: "9d9708fac90b8765127c65187f690b82"
server_key_version: "encryptionkey@0"
I20260806 23:57:27.132280 20149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 23:57:27.134537 20174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:27.135684 20149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:27.135809 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
format_stamp: "Formatted at 2026-08-06 23:57:27 on dist-test-slave-39l8"
server_key: "e79b9c5adf15ed697ad6dad50d048d10"
server_key_iv: "9d9708fac90b8765127c65187f690b82"
server_key_version: "encryptionkey@0"
I20260806 23:57:27.135910 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:27.155112 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:27.157670 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:27.157867 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:27.159250 20149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:27.159332 20149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:27.159417 20149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:27.159461 20149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:27.171298 20149 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:41091
I20260806 23:57:27.171644 20330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:41091 every 8 connection(s)
I20260806 23:57:27.172305 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:27.174577 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.160103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:51833 (local address 127.8.204.126:40461)
0806 23:57:27.160233 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.160237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.161320 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 23:57:27.162659 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.162663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.162764 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.162873 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.164123 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.165048 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.166258 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.166644 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.166808 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.170467 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:27.170475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:27.170479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:27.170512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:27.172199 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.172858 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.172862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.172864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.172960 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.173611 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.173614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.173615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.173809 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:27.173881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.174371 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.174463 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:57:27.175446 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.160596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.67:51833)
0806 23:57:27.161109 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.161158 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.162368 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.163049 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.163056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.163427 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:27.163951 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.163964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.165203 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.165206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.166104 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.166116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.166860 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.166865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.167087 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.167701 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:27.167721 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:27.170265 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:27.172353 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.172359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.172372 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.172734 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.173070 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.173074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.173077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.173460 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.173944 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:27.173950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:27.174163 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.174323 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.174571 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260806 23:57:27.176688 20331 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40461
I20260806 23:57:27.176846 20331 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:27.177614 20331 heartbeater.cc:511] Master 127.8.204.126:40461 requested a full tablet report, sending...
I20260806 23:57:27.178761 19623 ts_manager.cc:194] Registered new tserver with Master: dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:27.179477 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 20149
I20260806 23:57:27.179519 19623 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:51833
I20260806 23:57:27.179625 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:27.179934 9009 external_mini_cluster.cc:1428] Setting key cdb1b670f53fc74350fcf0ff272ea73a
I20260806 23:57:27.181844 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:57:27.186196 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.178761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.8.204.67:50463)
0806 23:57:27.179037 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.179051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.179146 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.179395 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.179398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.179593 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:27.179698 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.179705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.180662 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.180665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.181452 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.181462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.181989 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.181992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.182081 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.182636 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:27.182646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:27.183001 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:27.184802 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.184805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.184806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.184993 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.185287 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.185289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.185290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.185363 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.185793 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:27.185795 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:27.185867 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.185947 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.186048 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260806 23:57:27.186383 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.178831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:50463 (local address 127.8.204.126:40461)
0806 23:57:27.178940 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.178944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.179124 (+ 180us) server_negotiation.cc:407] Connection header received
0806 23:57:27.179236 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.179241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.179288 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.179369 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.179827 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.180456 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.181574 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.181873 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.181974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.183133 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:27.183137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:27.183139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:27.183163 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:27.184660 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.185090 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.185095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.185099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.185191 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.185450 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.185455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.185458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.185696 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:27.185803 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.186045 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.186167 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:57:27.207636 19690 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:27.201676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.8.204.126:40461)
0806 23:57:27.201813 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.201816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.201828 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:27.201865 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.201868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.201904 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.201996 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.202758 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.203293 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.204156 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.204367 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.204492 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.207520 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52240: BlockingRecv error: recv got EOF from 127.0.0.1:52240 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:27.216625 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.209140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.8.204.126:40461)
0806 23:57:27.209252 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.209255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.209416 (+ 161us) server_negotiation.cc:407] Connection header received
0806 23:57:27.209505 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.209507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.209543 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.209608 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.210156 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.210644 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.211509 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.211712 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.211785 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.213870 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:27.213873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:27.213889 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:27.213911 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:27.215364 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.215700 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.215703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.215704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.215775 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.215993 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.215995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.215996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.216154 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:27.216255 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.216437 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.216517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:57:27.219441 19623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:27.222838 19623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:27.237011 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.230910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41091 (local address 127.0.0.1:35656)
0806 23:57:27.231634 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.231680 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.231798 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.232856 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.232859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.232882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.232988 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.232995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.234371 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.234374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.235384 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.235395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236468 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.236582 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236617 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.236743 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.236852 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":1,"spinlock_wait_cycles":6016,"thread_start_us":67,"threads_started":1}
I20260806 23:57:27.237036 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.231525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.8.204.67:41091)
0806 23:57:27.232496 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.232504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.232521 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:57:27.232588 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.232592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.232717 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.232890 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.233099 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.234135 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.235512 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236323 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236454 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236631 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.236783 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.236885 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":126,"threads_started":1}
I20260806 23:57:27.237340 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.232010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.8.204.66:32911)
0806 23:57:27.232376 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.232387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.232984 (+ 597us) server_negotiation.cc:407] Connection header received
0806 23:57:27.233066 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.233072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.233225 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.233370 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.233608 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.234470 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.235861 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236424 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236530 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236683 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.237079 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.237205 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":134,"threads_started":1}
I20260806 23:57:27.237355 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.231285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.0.0.1:44978)
0806 23:57:27.232794 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.232806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.232893 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.233354 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.233357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.233374 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.233483 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.233490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.234600 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.234603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.235744 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.235752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236749 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.236830 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236849 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.236914 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.236982 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"mutex_wait_us":1,"spinlock_wait_cycles":3328,"thread_start_us":97,"threads_started":1}
I20260806 23:57:27.237668 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.231217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.8.204.65:45587)
0806 23:57:27.232094 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.232110 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.232134 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:57:27.232210 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.232222 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.232390 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.232550 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.232825 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.233982 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.235667 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236406 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236495 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236629 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.236992 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.237387 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":88,"threads_started":1}
I20260806 23:57:27.237990 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.231050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45587 (local address 127.0.0.1:53478)
0806 23:57:27.231634 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.231680 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.231798 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.232542 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.232548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.232570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.232687 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.232695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.234128 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.234132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.235543 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.235554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.236698 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.236701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.236806 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.236830 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.236899 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.237326 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":347,"spinlock_wait_cycles":3840,"thread_start_us":84,"threads_started":1}
I20260806 23:57:27.240326 20035 tablet_service.cc:1467] Processing CreateTablet for tablet ba27063c73754faca3247314b9de5748 (DEFAULT_TABLE table=test-table [id=95bb4c3573b744209a3822a9ed03c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.240350 19828 tablet_service.cc:1467] Processing CreateTablet for tablet ba27063c73754faca3247314b9de5748 (DEFAULT_TABLE table=test-table [id=95bb4c3573b744209a3822a9ed03c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.240484 20241 tablet_service.cc:1467] Processing CreateTablet for tablet ba27063c73754faca3247314b9de5748 (DEFAULT_TABLE table=test-table [id=95bb4c3573b744209a3822a9ed03c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.241364 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba27063c73754faca3247314b9de5748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.241384 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba27063c73754faca3247314b9de5748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.241384 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba27063c73754faca3247314b9de5748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.248070 20372 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Bootstrap starting.
I20260806 23:57:27.249861 20373 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Bootstrap starting.
I20260806 23:57:27.250332 20372 tablet_bootstrap.cc:654] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.251029 20374 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Bootstrap starting.
I20260806 23:57:27.251223 20372 log.cc:824] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:27.251670 20373 tablet_bootstrap.cc:654] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.252506 20373 log.cc:824] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:27.253002 20374 tablet_bootstrap.cc:654] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.253551 20372 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: No bootstrap required, opened a new log
I20260806 23:57:27.253756 20372 ts_tablet_manager.cc:1397] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:57:27.253805 20374 log.cc:824] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:27.254433 20373 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: No bootstrap required, opened a new log
I20260806 23:57:27.254617 20373 ts_tablet_manager.cc:1397] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:57:27.255777 20374 tablet_bootstrap.cc:492] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: No bootstrap required, opened a new log
I20260806 23:57:27.255954 20374 ts_tablet_manager.cc:1397] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:27.257112 20372 raft_consensus.cc:348] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.257334 20372 raft_consensus.cc:374] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.257387 20372 raft_consensus.cc:729] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.257805 20372 consensus_queue.cc:260] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.258450 20331 heartbeater.cc:503] Master 127.8.204.126:40461 was elected leader, sending a full tablet report...
I20260806 23:57:27.258327 20373 raft_consensus.cc:348] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.258559 20373 raft_consensus.cc:374] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.258606 20373 raft_consensus.cc:729] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.258821 20372 ts_tablet_manager.cc:1428] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:57:27.259148 20373 consensus_queue.cc:260] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.259718 19921 heartbeater.cc:503] Master 127.8.204.126:40461 was elected leader, sending a full tablet report...
I20260806 23:57:27.260098 20373 ts_tablet_manager.cc:1428] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 23:57:27.260156 20374 raft_consensus.cc:348] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.260381 20374 raft_consensus.cc:374] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.260418 20374 raft_consensus.cc:729] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.260793 20374 consensus_queue.cc:260] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.261346 20134 heartbeater.cc:503] Master 127.8.204.126:40461 was elected leader, sending a full tablet report...
I20260806 23:57:27.261673 20374 ts_tablet_manager.cc:1428] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:57:27.359629 20380 raft_consensus.cc:482] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:27.359838 20380 raft_consensus.cc:504] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.360977 20380 leader_election.cc:288] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091), 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911)
I20260806 23:57:27.364857 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.361201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.8.204.65:51077)
0806 23:57:27.361557 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.361568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.361684 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.361927 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.361930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.361945 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.362056 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.362061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.362825 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.362830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.363709 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.363719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.364374 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.364377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.364524 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.364545 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.364651 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.364718 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
I20260806 23:57:27.364845 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.361365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:51077 (local address 127.8.204.66:32911)
0806 23:57:27.361489 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.361494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.361649 (+ 155us) server_negotiation.cc:407] Connection header received
0806 23:57:27.361795 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.361798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.361842 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.361916 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.362161 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.362670 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.363839 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.364242 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.364374 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.364426 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.364668 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.364731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:27.364845 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.361251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:45507 (local address 127.8.204.67:41091)
0806 23:57:27.361480 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.361487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.361576 (+ 89us) server_negotiation.cc:407] Connection header received
0806 23:57:27.361741 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.361743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.361782 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.361859 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.362110 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.362637 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.363841 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.364235 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.364369 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.364426 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.364666 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.364740 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260806 23:57:27.364871 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.361160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41091 (local address 127.8.204.65:45507)
0806 23:57:27.361477 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.361491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.361628 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.361872 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.361874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.361889 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.362001 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.362007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.362800 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.362805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.363713 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.363723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.364377 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.364381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.364533 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.364551 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.364648 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.364713 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":98,"thread_start_us":103,"threads_started":1}
I20260806 23:57:27.365566 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba27063c73754faca3247314b9de5748" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" is_pre_election: true
I20260806 23:57:27.365576 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba27063c73754faca3247314b9de5748" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971" is_pre_election: true
I20260806 23:57:27.365836 20265 raft_consensus.cc:2454] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:27.365830 20062 raft_consensus.cc:2454] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:27.366297 19748 leader_election.cc:302] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:27.366564 20380 raft_consensus.cc:2781] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:27.366644 20380 raft_consensus.cc:482] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:27.366707 20380 raft_consensus.cc:3037] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.367738 20380 raft_consensus.cc:504] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.368013 20380 leader_election.cc:288] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Requested vote from peers dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091), 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911)
I20260806 23:57:27.368374 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba27063c73754faca3247314b9de5748" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971"
I20260806 23:57:27.368374 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba27063c73754faca3247314b9de5748" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
I20260806 23:57:27.368490 20062 raft_consensus.cc:3037] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.368489 20265 raft_consensus.cc:3037] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.369714 20265 raft_consensus.cc:2454] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:27.369714 20062 raft_consensus.cc:2454] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:27.369982 19748 leader_election.cc:302] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:27.370154 20380 raft_consensus.cc:2781] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:27.370357 20380 raft_consensus.cc:686] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Becoming Leader. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Running, Role: LEADER
I20260806 23:57:27.370684 20380 consensus_queue.cc:237] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } }
I20260806 23:57:27.373821 19622 catalog_manager.cc:5495] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3bbff8ace5944f3841e2579e2210145 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "f3bbff8ace5944f3841e2579e2210145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:27.385762 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.382196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.8.204.65:45587)
0806 23:57:27.382329 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.382335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.382365 (+ 30us) server_negotiation.cc:407] Connection header received
0806 23:57:27.382537 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.382542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.382595 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.382700 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:27.382958 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.383547 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.384369 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.384649 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.384766 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.385033 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.385421 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.385537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.385600 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.385648 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:27.390960 20265 raft_consensus.cc:1261] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.391242 20062 raft_consensus.cc:1261] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.391609 20385 consensus_queue.cc:1035] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.391798 20380 consensus_queue.cc:1035] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.400053 20392 mvcc.cc:205] Tried to move back new op lower bound from 7315704411707236352 to 7315704411633352704. Current Snapshot: MvccSnapshot[applied={T|T < 7315704411707236352}]
I20260806 23:57:27.401415 20391 mvcc.cc:205] Tried to move back new op lower bound from 7315704411707236352 to 7315704411633352704. Current Snapshot: MvccSnapshot[applied={T|T < 7315704411707236352}]
I20260806 23:57:27.407209 19621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:27.407366 19621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:27.409770 19621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Sending DeleteTablet for 3 replicas of tablet ba27063c73754faca3247314b9de5748
I20260806 23:57:27.410529 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet ba27063c73754faca3247314b9de5748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35656
I20260806 23:57:27.411088 19828 tablet_service.cc:1514] Processing DeleteTablet for tablet ba27063c73754faca3247314b9de5748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53478
I20260806 23:57:27.411134 20035 tablet_service.cc:1514] Processing DeleteTablet for tablet ba27063c73754faca3247314b9de5748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44978
I20260806 23:57:27.411595 20406 tablet_replica.cc:330] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: stopping tablet replica
I20260806 23:57:27.411556 19622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:57:27.411883 20406 raft_consensus.cc:2229] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 23:57:27.411950 19622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:27.412223 20406 raft_consensus.cc:2258] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.413275 20407 tablet_replica.cc:330] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: stopping tablet replica
I20260806 23:57:27.413533 20407 raft_consensus.cc:2229] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:27.413755 20406 ts_tablet_manager.cc:1905] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.413857 20407 raft_consensus.cc:2258] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.414165 20408 tablet_replica.cc:330] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: stopping tablet replica
I20260806 23:57:27.414392 20408 raft_consensus.cc:2229] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:27.414621 20408 raft_consensus.cc:2258] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.416499 20406 ts_tablet_manager.cc:1918] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:27.416611 20406 log.cc:1196] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/ba27063c73754faca3247314b9de5748
I20260806 23:57:27.416774 20407 ts_tablet_manager.cc:1905] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.417299 20241 tablet_service.cc:1467] Processing CreateTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (DEFAULT_TABLE table=test-table [id=5aa6f654405e478faf2b23acde5c346e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.417352 19828 tablet_service.cc:1467] Processing CreateTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (DEFAULT_TABLE table=test-table [id=5aa6f654405e478faf2b23acde5c346e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.417569 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b22baa3c1524a4aaad5804d2e2c4a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.417604 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b22baa3c1524a4aaad5804d2e2c4a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.419693 20372 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Bootstrap starting.
I20260806 23:57:27.419734 20373 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Bootstrap starting.
I20260806 23:57:27.420492 20035 tablet_service.cc:1467] Processing CreateTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (DEFAULT_TABLE table=test-table [id=5aa6f654405e478faf2b23acde5c346e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.420727 20372 tablet_bootstrap.cc:654] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.420737 20373 tablet_bootstrap.cc:654] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.420770 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b22baa3c1524a4aaad5804d2e2c4a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.421650 20406 ts_tablet_manager.cc:1939] T ba27063c73754faca3247314b9de5748 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting consensus metadata
I20260806 23:57:27.421790 20408 ts_tablet_manager.cc:1905] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.422823 19601 catalog_manager.cc:4841] TS dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091): tablet ba27063c73754faca3247314b9de5748 (table test-table [id=95bb4c3573b744209a3822a9ed03c107]) successfully deleted
I20260806 23:57:27.423512 20374 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Bootstrap starting.
I20260806 23:57:27.423673 20372 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: No bootstrap required, opened a new log
I20260806 23:57:27.423753 20372 ts_tablet_manager.cc:1397] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 23:57:27.423866 20407 ts_tablet_manager.cc:1918] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:27.423943 20407 log.cc:1196] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/ba27063c73754faca3247314b9de5748
I20260806 23:57:27.424256 20407 ts_tablet_manager.cc:1939] T ba27063c73754faca3247314b9de5748 P f3bbff8ace5944f3841e2579e2210145: Deleting consensus metadata
I20260806 23:57:27.424261 20372 raft_consensus.cc:348] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.424367 20372 raft_consensus.cc:374] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.424408 20372 raft_consensus.cc:729] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.424429 20374 tablet_bootstrap.cc:654] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.424530 20372 consensus_queue.cc:260] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.424772 20372 ts_tablet_manager.cc:1428] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:27.425236 19600 catalog_manager.cc:4841] TS f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587): tablet ba27063c73754faca3247314b9de5748 (table test-table [id=95bb4c3573b744209a3822a9ed03c107]) successfully deleted
I20260806 23:57:27.426221 20374 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: No bootstrap required, opened a new log
I20260806 23:57:27.426293 20374 ts_tablet_manager.cc:1397] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:57:27.426715 20374 raft_consensus.cc:348] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.426821 20374 raft_consensus.cc:374] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.426848 20374 raft_consensus.cc:729] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.426942 20374 consensus_queue.cc:260] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.427150 20374 ts_tablet_manager.cc:1428] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:57:27.427868 20408 ts_tablet_manager.cc:1918] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:27.427991 20408 log.cc:1196] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/ba27063c73754faca3247314b9de5748
I20260806 23:57:27.428349 20408 ts_tablet_manager.cc:1939] T ba27063c73754faca3247314b9de5748 P 3d392a0e48b24b09ac002babde098971: Deleting consensus metadata
I20260806 23:57:27.429989 19599 catalog_manager.cc:4841] TS 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911): tablet ba27063c73754faca3247314b9de5748 (table test-table [id=95bb4c3573b744209a3822a9ed03c107]) successfully deleted
I20260806 23:57:27.431383 20373 tablet_bootstrap.cc:492] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: No bootstrap required, opened a new log
I20260806 23:57:27.431471 20373 ts_tablet_manager.cc:1397] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260806 23:57:27.431875 20373 raft_consensus.cc:348] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.431993 20373 raft_consensus.cc:374] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.432051 20373 raft_consensus.cc:729] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.432173 20373 consensus_queue.cc:260] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.432435 20373 ts_tablet_manager.cc:1428] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260806 23:57:27.458833 20135 tablet.cc:2369] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:57:27.501578 19922 tablet.cc:2369] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:27.669986 20396 raft_consensus.cc:482] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:27.670171 20396 raft_consensus.cc:504] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.670636 20396 leader_election.cc:288] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:27.671113 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b22baa3c1524a4aaad5804d2e2c4a76" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971" is_pre_election: true
I20260806 23:57:27.671118 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b22baa3c1524a4aaad5804d2e2c4a76" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" is_pre_election: true
I20260806 23:57:27.671284 20062 raft_consensus.cc:2454] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:27.671295 20265 raft_consensus.cc:2454] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:27.671618 19752 leader_election.cc:302] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc85758b3bef4f0c82f62e3bdd8d44a3, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:27.671814 20396 raft_consensus.cc:2781] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:27.671878 20396 raft_consensus.cc:482] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:27.671952 20396 raft_consensus.cc:3037] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.672976 20396 raft_consensus.cc:504] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.673332 20396 leader_election.cc:288] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Requested vote from peers 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
W20260806 23:57:27.673556 20334 tablet.cc:2369] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:27.673703 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b22baa3c1524a4aaad5804d2e2c4a76" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
I20260806 23:57:27.673751 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b22baa3c1524a4aaad5804d2e2c4a76" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971"
I20260806 23:57:27.673839 20265 raft_consensus.cc:3037] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.673848 20062 raft_consensus.cc:3037] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.674770 20265 raft_consensus.cc:2454] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:27.674783 20062 raft_consensus.cc:2454] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:27.675055 19752 leader_election.cc:302] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc85758b3bef4f0c82f62e3bdd8d44a3, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:27.675249 20396 raft_consensus.cc:2781] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:27.675354 20396 raft_consensus.cc:686] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Becoming Leader. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Running, Role: LEADER
I20260806 23:57:27.675498 20396 consensus_queue.cc:237] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.677263 19622 catalog_manager.cc:5495] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3bbff8ace5944f3841e2579e2210145 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "f3bbff8ace5944f3841e2579e2210145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060646, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.126@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:27.735611 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.728130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40461 (local address 127.0.0.1:52260)
0806 23:57:27.728418 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.728452 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.728593 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.728839 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.728842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.729228 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:27.729306 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.729312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.730012 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.730017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.730508 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.730516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.730982 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.730985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.731050 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.731540 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:27.731550 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:27.732922 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:27.734304 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.734311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.734315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.734516 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.734803 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:27.734805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:27.734806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:27.734877 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:27.735184 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:27.735193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:27.735284 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.735363 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.735452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260806 23:57:27.735632 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.728188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.8.204.126:40461)
0806 23:57:27.728424 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.728453 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.728537 (+ 84us) server_negotiation.cc:407] Connection header received
0806 23:57:27.728695 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.728698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.728738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.728823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.729416 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.729912 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.730621 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.730879 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.730982 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.733056 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:27.733059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:27.733061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:27.733085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:27.734215 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.734598 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.734600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.734601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.734668 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.734949 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.734951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.734952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.735102 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:27.735170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.735415 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.735497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":111,"threads_started":1}
I20260806 23:57:27.738893 19622 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:52260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:57:27.739557 19622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:27.743772 19828 tablet_service.cc:1467] Processing CreateTablet for tablet a879d4114a1f4679a28a09a0e4dd1fc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27ce83c0c6c24248b5c0c7e6cf1f9848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:57:27.743800 20035 tablet_service.cc:1467] Processing CreateTablet for tablet a879d4114a1f4679a28a09a0e4dd1fc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27ce83c0c6c24248b5c0c7e6cf1f9848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:57:27.744001 20241 tablet_service.cc:1467] Processing CreateTablet for tablet a879d4114a1f4679a28a09a0e4dd1fc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27ce83c0c6c24248b5c0c7e6cf1f9848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:57:27.744148 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a879d4114a1f4679a28a09a0e4dd1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.744150 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a879d4114a1f4679a28a09a0e4dd1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.744323 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a879d4114a1f4679a28a09a0e4dd1fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.746803 20373 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: Bootstrap starting.
I20260806 23:57:27.746886 20374 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: Bootstrap starting.
I20260806 23:57:27.747516 20373 tablet_bootstrap.cc:654] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.747576 20374 tablet_bootstrap.cc:654] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.748458 20372 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: Bootstrap starting.
I20260806 23:57:27.748744 20374 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: No bootstrap required, opened a new log
I20260806 23:57:27.748835 20374 ts_tablet_manager.cc:1397] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 23:57:27.748888 20373 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: No bootstrap required, opened a new log
I20260806 23:57:27.748986 20373 ts_tablet_manager.cc:1397] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:27.749313 20372 tablet_bootstrap.cc:654] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.749264 20374 raft_consensus.cc:348] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.749379 20374 raft_consensus.cc:374] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.749411 20374 raft_consensus.cc:729] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.749521 20374 consensus_queue.cc:260] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.749550 20373 raft_consensus.cc:348] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.749708 20373 raft_consensus.cc:374] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.749742 20373 raft_consensus.cc:729] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.749814 20374 ts_tablet_manager.cc:1428] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:57:27.749852 20373 consensus_queue.cc:260] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.750156 20381 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.750278 20381 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.750520 20396 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.750677 20396 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.751058 20372 tablet_bootstrap.cc:492] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: No bootstrap required, opened a new log
I20260806 23:57:27.751187 20372 ts_tablet_manager.cc:1397] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 23:57:27.751678 20372 raft_consensus.cc:348] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.751781 20372 raft_consensus.cc:374] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.751813 20372 raft_consensus.cc:729] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.751938 20372 consensus_queue.cc:260] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.752265 20372 ts_tablet_manager.cc:1428] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:57:27.752230 20379 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.752352 20379 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.753216 20373 ts_tablet_manager.cc:1428] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 23:57:27.800204 20381 raft_consensus.cc:482] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:27.800318 20381 raft_consensus.cc:504] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.801326 20381 leader_election.cc:288] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:27.804924 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.801654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:42179 (local address 127.8.204.65:45587)
0806 23:57:27.801763 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.801768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.801896 (+ 128us) server_negotiation.cc:407] Connection header received
0806 23:57:27.802053 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.802062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.802100 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.802175 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.802487 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.802997 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804058 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.804411 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.804583 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.804763 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.804824 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 23:57:27.804946 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.801636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:32823 (local address 127.8.204.67:41091)
0806 23:57:27.801757 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.801761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.801916 (+ 155us) server_negotiation.cc:407] Connection header received
0806 23:57:27.802043 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.802048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.802096 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.802175 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:27.802479 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.803025 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804070 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.804429 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804545 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.804592 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.804781 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.804844 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:27.804970 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.801539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41091 (local address 127.8.204.66:32823)
0806 23:57:27.801829 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.801839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.801948 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.802187 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.802190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.802204 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.802363 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.802368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.803140 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.803146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.803946 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.803956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.804543 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.804657 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.804682 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.804764 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.804826 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":44,"spinlock_wait_cycles":8960,"thread_start_us":93,"threads_started":1}
I20260806 23:57:27.804950 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.801493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45587 (local address 127.8.204.66:42179)
0806 23:57:27.801815 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.801825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.801935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.802195 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.802198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.802213 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:27.802373 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.802378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.803110 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.803115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.803938 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.803946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.804526 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.804530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.804617 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.804656 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.804732 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.804799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":9,"thread_start_us":96,"threads_started":1}
I20260806 23:57:27.805361 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a879d4114a1f4679a28a09a0e4dd1fc2" candidate_uuid: "3d392a0e48b24b09ac002babde098971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" is_pre_election: true
I20260806 23:57:27.805505 20265 raft_consensus.cc:2454] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392a0e48b24b09ac002babde098971 in term 0.
I20260806 23:57:27.805680 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a879d4114a1f4679a28a09a0e4dd1fc2" candidate_uuid: "3d392a0e48b24b09ac002babde098971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bbff8ace5944f3841e2579e2210145" is_pre_election: true
I20260806 23:57:27.805843 19959 leader_election.cc:302] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, dc85758b3bef4f0c82f62e3bdd8d44a3; no voters:
I20260806 23:57:27.805891 19855 raft_consensus.cc:2454] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392a0e48b24b09ac002babde098971 in term 0.
I20260806 23:57:27.806088 20381 raft_consensus.cc:2781] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:27.806145 20381 raft_consensus.cc:482] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:27.806198 20381 raft_consensus.cc:3037] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.807130 20381 raft_consensus.cc:504] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.807484 20381 leader_election.cc:288] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [CANDIDATE]: Term 1 election: Requested vote from peers f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:27.807780 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a879d4114a1f4679a28a09a0e4dd1fc2" candidate_uuid: "3d392a0e48b24b09ac002babde098971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bbff8ace5944f3841e2579e2210145"
I20260806 23:57:27.807776 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a879d4114a1f4679a28a09a0e4dd1fc2" candidate_uuid: "3d392a0e48b24b09ac002babde098971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
I20260806 23:57:27.807909 19855 raft_consensus.cc:3037] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.807963 20265 raft_consensus.cc:3037] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:27.808967 20265 raft_consensus.cc:2454] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392a0e48b24b09ac002babde098971 in term 1.
I20260806 23:57:27.809109 19855 raft_consensus.cc:2454] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392a0e48b24b09ac002babde098971 in term 1.
I20260806 23:57:27.809243 19959 leader_election.cc:302] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, dc85758b3bef4f0c82f62e3bdd8d44a3; no voters:
I20260806 23:57:27.809461 20381 raft_consensus.cc:2781] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:27.809634 20381 raft_consensus.cc:686] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [term 1 LEADER]: Becoming Leader. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Running, Role: LEADER
I20260806 23:57:27.809880 20381 consensus_queue.cc:237] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.811127 20433 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: TxnStatusTablet state changed. Reason: New leader 3d392a0e48b24b09ac002babde098971. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.811244 20433 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:57:27.811510 20435 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:57:27.811659 20435 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:57:27.812240 19623 catalog_manager.cc:5495] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d392a0e48b24b09ac002babde098971 (127.8.204.66). New cstate: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:27.834443 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.830584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.0.0.1:44992)
0806 23:57:27.830744 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.830760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.830847 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.831151 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.831153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.831158 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:57:27.831244 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.831247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.832147 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.832150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.832735 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.832743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.833288 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.833291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.833351 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.833392 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.834092 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.834129 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.834199 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.834286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 23:57:27.834832 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.830695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.8.204.66:32911)
0806 23:57:27.830823 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.830828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.830841 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:27.830980 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.830983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.831036 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.831129 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:27.831332 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.832039 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.832842 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.833179 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.833285 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.833547 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.833989 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.834112 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.834604 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.834665 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 23:57:27.837273 20265 raft_consensus.cc:1261] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Refusing update from remote peer 3d392a0e48b24b09ac002babde098971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.837422 19855 raft_consensus.cc:1261] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Refusing update from remote peer 3d392a0e48b24b09ac002babde098971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.837733 20433 consensus_queue.cc:1035] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.837878 20381 consensus_queue.cc:1035] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.839457 20396 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: TxnStatusTablet state changed. Reason: New leader 3d392a0e48b24b09ac002babde098971. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.839491 20379 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: TxnStatusTablet state changed. Reason: New leader 3d392a0e48b24b09ac002babde098971. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.839577 20396 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.839589 20379 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.843914 20379 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.844002 20379 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P dc85758b3bef4f0c82f62e3bdd8d44a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.844208 20396 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.844306 20396 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P f3bbff8ace5944f3841e2579e2210145: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:57:27.844383 20388 mvcc.cc:205] Tried to move back new op lower bound from 7315704413536260096 to 7315704413430771712. Current Snapshot: MvccSnapshot[applied={T|T < 7315704413536260096 or (T in {7315704413536260096})}]
I20260806 23:57:27.842841 20440 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.846616 20440 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:57:27.842944 20381 tablet_replica.cc:439] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d392a0e48b24b09ac002babde098971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } } }
I20260806 23:57:27.846870 20381 tablet_replica.cc:442] T a879d4114a1f4679a28a09a0e4dd1fc2 P 3d392a0e48b24b09ac002babde098971: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:57:27.857475 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.854260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:56743 (local address 127.8.204.66:32911)
0806 23:57:27.854362 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.854367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.854520 (+ 153us) server_negotiation.cc:407] Connection header received
0806 23:57:27.854637 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.854639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.854673 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.854805 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:27.855017 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.855469 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.856585 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.856774 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.856845 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.857132 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.857193 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.857244 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.857356 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.857391 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:27.857553 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.854142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.8.204.65:56743)
0806 23:57:27.854439 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.854452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.854551 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.854816 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.854819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.854826 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:57:27.854936 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.854941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.855574 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.855577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.856476 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.856487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.856874 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.856878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.856965 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.857014 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.857269 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.857276 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.857339 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.857418 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260806 23:57:27.863675 20265 raft_consensus.cc:1261] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.863682 20062 raft_consensus.cc:1261] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:27.864034 20380 consensus_queue.cc:1035] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.864173 20396 consensus_queue.cc:1035] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:27.885170 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.881500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45587 (local address 127.8.204.66:56129)
0806 23:57:27.881810 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:27.881825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:27.881922 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:27.882176 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:27.882179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:27.882186 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:57:27.882361 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.882372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.883189 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.883195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.884032 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:27.884043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.884477 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.884480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:27.884569 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.884619 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.884910 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.884919 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 23:57:27.884978 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 23:57:27.885047 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
I20260806 23:57:27.885156 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.881613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:56129 (local address 127.8.204.65:45587)
0806 23:57:27.881723 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.881727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.881876 (+ 149us) server_negotiation.cc:407] Connection header received
0806 23:57:27.882025 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.882028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.882080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.882153 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:27.882511 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.883076 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.884153 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.884366 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.884474 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.884771 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:27.884832 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:27.884906 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.884987 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.885032 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:27.910293 19623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:27.910445 19623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:27.912335 19623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Sending DeleteTablet for 3 replicas of tablet 8b22baa3c1524a4aaad5804d2e2c4a76
I20260806 23:57:27.912838 19828 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53478
I20260806 23:57:27.912925 20035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44978
I20260806 23:57:27.913040 20407 tablet_replica.cc:330] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: stopping tablet replica
I20260806 23:57:27.913182 20407 raft_consensus.cc:2229] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:27.913305 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b22baa3c1524a4aaad5804d2e2c4a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:27 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35656
I20260806 23:57:27.913378 20407 raft_consensus.cc:2258] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.913486 20406 tablet_replica.cc:330] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: stopping tablet replica
I20260806 23:57:27.913545 20408 tablet_replica.cc:330] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: stopping tablet replica
I20260806 23:57:27.913600 20406 raft_consensus.cc:2229] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:27.913652 20408 raft_consensus.cc:2229] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:27.913705 20406 raft_consensus.cc:2258] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.913735 20408 raft_consensus.cc:2258] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:27.914494 20406 ts_tablet_manager.cc:1905] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.914515 20408 ts_tablet_manager.cc:1905] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.917315 20406 ts_tablet_manager.cc:1918] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:27.917382 20406 log.cc:1196] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/8b22baa3c1524a4aaad5804d2e2c4a76
I20260806 23:57:27.917650 20406 ts_tablet_manager.cc:1939] T 8b22baa3c1524a4aaad5804d2e2c4a76 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting consensus metadata
I20260806 23:57:27.917706 20408 ts_tablet_manager.cc:1918] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:27.917784 20408 log.cc:1196] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/8b22baa3c1524a4aaad5804d2e2c4a76
I20260806 23:57:27.918121 20408 ts_tablet_manager.cc:1939] T 8b22baa3c1524a4aaad5804d2e2c4a76 P 3d392a0e48b24b09ac002babde098971: Deleting consensus metadata
I20260806 23:57:27.918464 19601 catalog_manager.cc:4841] TS dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091): tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (table test-table [id=5aa6f654405e478faf2b23acde5c346e]) successfully deleted
I20260806 23:57:27.918792 19599 catalog_manager.cc:4841] TS 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911): tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (table test-table [id=5aa6f654405e478faf2b23acde5c346e]) successfully deleted
I20260806 23:57:27.919459 20407 ts_tablet_manager.cc:1905] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:27.922109 20407 ts_tablet_manager.cc:1918] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:27.922194 20407 log.cc:1196] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/8b22baa3c1524a4aaad5804d2e2c4a76
I20260806 23:57:27.922472 20407 ts_tablet_manager.cc:1939] T 8b22baa3c1524a4aaad5804d2e2c4a76 P f3bbff8ace5944f3841e2579e2210145: Deleting consensus metadata
I20260806 23:57:27.923223 19600 catalog_manager.cc:4841] TS f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587): tablet 8b22baa3c1524a4aaad5804d2e2c4a76 (table test-table [id=5aa6f654405e478faf2b23acde5c346e]) successfully deleted
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:57:27.943318 20426 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:27.936934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.8.204.126:40461)
0806 23:57:27.937101 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.937106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.937271 (+ 165us) server_negotiation.cc:407] Connection header received
0806 23:57:27.937433 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.937436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.937488 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.937573 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.938308 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.939023 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.939928 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.940164 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.940295 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.943197 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52270: BlockingRecv error: recv got EOF from 127.0.0.1:52270 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 23:57:27 dist-test-slave-39l8 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM
I20260806 23:57:27.952837 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:27.944977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.8.204.126:40461)
0806 23:57:27.945090 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:27.945093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:27.945275 (+ 182us) server_negotiation.cc:407] Connection header received
0806 23:57:27.945406 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:27.945409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:27.945446 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:27.945526 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:27.946064 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.946564 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.947404 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:27.947632 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:27.947732 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:27.949814 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:27.949818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:27.949821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:27.949844 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:27.951216 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.951623 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.951627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.951628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.951710 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:27.951992 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:27.951996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:27.951997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:27.952161 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:27.952327 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:27.952567 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 23:57:27.952681 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:57:27.955526 19623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:27.955943 19623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:27.960940 19828 tablet_service.cc:1467] Processing CreateTablet for tablet c4a045c74ae541d98eece00949f070e9 (DEFAULT_TABLE table=test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.961108 20035 tablet_service.cc:1467] Processing CreateTablet for tablet c4a045c74ae541d98eece00949f070e9 (DEFAULT_TABLE table=test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.961251 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a045c74ae541d98eece00949f070e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.961370 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a045c74ae541d98eece00949f070e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.961483 20241 tablet_service.cc:1467] Processing CreateTablet for tablet c4a045c74ae541d98eece00949f070e9 (DEFAULT_TABLE table=test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:27.961742 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a045c74ae541d98eece00949f070e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:27.963696 20374 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Bootstrap starting.
I20260806 23:57:27.964047 20373 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Bootstrap starting.
I20260806 23:57:27.964577 20374 tablet_bootstrap.cc:654] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.964888 20373 tablet_bootstrap.cc:654] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.965382 20372 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Bootstrap starting.
I20260806 23:57:27.966128 20374 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: No bootstrap required, opened a new log
I20260806 23:57:27.966208 20374 ts_tablet_manager.cc:1397] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 23:57:27.966298 20372 tablet_bootstrap.cc:654] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:27.966579 20374 raft_consensus.cc:348] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.966687 20374 raft_consensus.cc:374] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.966739 20374 raft_consensus.cc:729] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.966840 20374 consensus_queue.cc:260] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.967063 20374 ts_tablet_manager.cc:1428] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:27.969252 20372 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: No bootstrap required, opened a new log
I20260806 23:57:27.969331 20372 ts_tablet_manager.cc:1397] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 23:57:27.969729 20372 raft_consensus.cc:348] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.969818 20372 raft_consensus.cc:374] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.969844 20372 raft_consensus.cc:729] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.969955 20372 consensus_queue.cc:260] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.970162 20372 ts_tablet_manager.cc:1428] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:27.975082 20373 tablet_bootstrap.cc:492] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: No bootstrap required, opened a new log
I20260806 23:57:27.975198 20373 ts_tablet_manager.cc:1397] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260806 23:57:27.975600 20373 raft_consensus.cc:348] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.975718 20373 raft_consensus.cc:374] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:27.975761 20373 raft_consensus.cc:729] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Initialized, Role: FOLLOWER
I20260806 23:57:27.975886 20373 consensus_queue.cc:260] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:27.976142 20373 ts_tablet_manager.cc:1428] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:28.194615 20396 raft_consensus.cc:482] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:28.194767 20396 raft_consensus.cc:504] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.195223 20396 leader_election.cc:288] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:28.195710 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a045c74ae541d98eece00949f070e9" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" is_pre_election: true
I20260806 23:57:28.195736 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a045c74ae541d98eece00949f070e9" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971" is_pre_election: true
I20260806 23:57:28.195832 20264 raft_consensus.cc:2454] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:28.195930 20062 raft_consensus.cc:2454] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 0.
I20260806 23:57:28.196219 19752 leader_election.cc:302] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc85758b3bef4f0c82f62e3bdd8d44a3, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:28.196432 20396 raft_consensus.cc:2781] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:28.196491 20396 raft_consensus.cc:482] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:28.196518 20396 raft_consensus.cc:3037] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.197449 20396 raft_consensus.cc:504] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.197757 20396 leader_election.cc:288] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Requested vote from peers 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911), dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091)
I20260806 23:57:28.198091 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a045c74ae541d98eece00949f070e9" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971"
I20260806 23:57:28.198118 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a045c74ae541d98eece00949f070e9" candidate_uuid: "f3bbff8ace5944f3841e2579e2210145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3"
I20260806 23:57:28.198213 20264 raft_consensus.cc:3037] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.198225 20062 raft_consensus.cc:3037] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.199361 20264 raft_consensus.cc:2454] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:28.199361 20062 raft_consensus.cc:2454] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3bbff8ace5944f3841e2579e2210145 in term 1.
I20260806 23:57:28.199651 19752 leader_election.cc:302] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc85758b3bef4f0c82f62e3bdd8d44a3, f3bbff8ace5944f3841e2579e2210145; no voters:
I20260806 23:57:28.199839 20396 raft_consensus.cc:2781] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:28.199939 20396 raft_consensus.cc:686] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Becoming Leader. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Running, Role: LEADER
I20260806 23:57:28.200083 20396 consensus_queue.cc:237] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.201673 19623 catalog_manager.cc:5495] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3bbff8ace5944f3841e2579e2210145 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "f3bbff8ace5944f3841e2579e2210145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:28.212055 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.209286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.8.204.65:45587)
0806 23:57:28.209396 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.209400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.209444 (+ 44us) server_negotiation.cc:407] Connection header received
0806 23:57:28.209565 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.209567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.209608 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.209668 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:28.209875 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.210339 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.211256 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.211439 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.211512 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.211720 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.211775 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.211824 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.211898 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.211941 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:57:28.213798 20264 raft_consensus.cc:1261] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:28.213798 20062 raft_consensus.cc:1261] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Refusing update from remote peer f3bbff8ace5944f3841e2579e2210145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:28.214159 20380 consensus_queue.cc:1035] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:28.214303 20396 consensus_queue.cc:1035] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:28.221071 19622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:28.221194 19622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:28.223042 19622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Sending DeleteTablet for 3 replicas of tablet c4a045c74ae541d98eece00949f070e9
I20260806 23:57:28.223564 19828 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a045c74ae541d98eece00949f070e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53478
I20260806 23:57:28.223564 20035 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a045c74ae541d98eece00949f070e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44978
I20260806 23:57:28.223745 20407 tablet_replica.cc:330] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: stopping tablet replica
I20260806 23:57:28.223778 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a045c74ae541d98eece00949f070e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35656
I20260806 23:57:28.223855 20407 raft_consensus.cc:2229] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:28.223941 20408 tablet_replica.cc:330] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: stopping tablet replica
I20260806 23:57:28.223951 20406 tablet_replica.cc:330] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: stopping tablet replica
I20260806 23:57:28.224022 20407 raft_consensus.cc:2258] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.224031 20408 raft_consensus.cc:2229] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:28.224061 20406 raft_consensus.cc:2229] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:28.224148 20408 raft_consensus.cc:2258] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.224177 20406 raft_consensus.cc:2258] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.224725 20407 ts_tablet_manager.cc:1905] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:28.224987 20408 ts_tablet_manager.cc:1905] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:28.225046 20406 ts_tablet_manager.cc:1905] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:28.227121 20408 ts_tablet_manager.cc:1918] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:28.227216 20408 log.cc:1196] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/c4a045c74ae541d98eece00949f070e9
I20260806 23:57:28.227269 20407 ts_tablet_manager.cc:1918] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:28.227360 20407 log.cc:1196] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/c4a045c74ae541d98eece00949f070e9
I20260806 23:57:28.227450 20406 ts_tablet_manager.cc:1918] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:28.227536 20406 log.cc:1196] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/c4a045c74ae541d98eece00949f070e9
I20260806 23:57:28.227545 20408 ts_tablet_manager.cc:1939] T c4a045c74ae541d98eece00949f070e9 P 3d392a0e48b24b09ac002babde098971: Deleting consensus metadata
I20260806 23:57:28.227677 20407 ts_tablet_manager.cc:1939] T c4a045c74ae541d98eece00949f070e9 P f3bbff8ace5944f3841e2579e2210145: Deleting consensus metadata
I20260806 23:57:28.227795 20406 ts_tablet_manager.cc:1939] T c4a045c74ae541d98eece00949f070e9 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting consensus metadata
I20260806 23:57:28.228173 19599 catalog_manager.cc:4841] TS 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911): tablet c4a045c74ae541d98eece00949f070e9 (table test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]) successfully deleted
I20260806 23:57:28.228332 19600 catalog_manager.cc:4841] TS f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587): tablet c4a045c74ae541d98eece00949f070e9 (table test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]) successfully deleted
I20260806 23:57:28.228392 19601 catalog_manager.cc:4841] TS dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091): tablet c4a045c74ae541d98eece00949f070e9 (table test-table [id=65707dd1a41d4e08ae1b5bd6dadccbc0]) successfully deleted
I20260806 23:57:28.228464 19622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:28.228803 19622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:28.232637 19828 tablet_service.cc:1467] Processing CreateTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 (DEFAULT_TABLE table=test-table [id=861db7ec4d674aaaa2924897c4d89d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:28.232692 20035 tablet_service.cc:1467] Processing CreateTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 (DEFAULT_TABLE table=test-table [id=861db7ec4d674aaaa2924897c4d89d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:28.232777 20241 tablet_service.cc:1467] Processing CreateTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 (DEFAULT_TABLE table=test-table [id=861db7ec4d674aaaa2924897c4d89d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:28.232903 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bc812cc5d04ea3bb35d29bd29a2813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:28.232969 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bc812cc5d04ea3bb35d29bd29a2813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:28.232992 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bc812cc5d04ea3bb35d29bd29a2813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:28.235066 20374 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Bootstrap starting.
I20260806 23:57:28.235145 20373 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: Bootstrap starting.
I20260806 23:57:28.235893 20374 tablet_bootstrap.cc:654] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:28.235932 20373 tablet_bootstrap.cc:654] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:28.236310 20372 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Bootstrap starting.
I20260806 23:57:28.237006 20372 tablet_bootstrap.cc:654] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:28.237147 20373 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: No bootstrap required, opened a new log
I20260806 23:57:28.237191 20374 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: No bootstrap required, opened a new log
I20260806 23:57:28.237251 20373 ts_tablet_manager.cc:1397] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:28.237263 20374 ts_tablet_manager.cc:1397] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:28.237620 20374 raft_consensus.cc:348] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.237632 20373 raft_consensus.cc:348] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.237740 20374 raft_consensus.cc:374] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:28.237748 20373 raft_consensus.cc:374] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:28.237778 20374 raft_consensus.cc:729] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392a0e48b24b09ac002babde098971, State: Initialized, Role: FOLLOWER
I20260806 23:57:28.237782 20373 raft_consensus.cc:729] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bbff8ace5944f3841e2579e2210145, State: Initialized, Role: FOLLOWER
I20260806 23:57:28.237893 20373 consensus_queue.cc:260] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.237907 20374 consensus_queue.cc:260] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.238135 20373 ts_tablet_manager.cc:1428] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:28.238163 20372 tablet_bootstrap.cc:492] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: No bootstrap required, opened a new log
I20260806 23:57:28.238152 20374 ts_tablet_manager.cc:1428] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:57:28.238240 20372 ts_tablet_manager.cc:1397] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:28.238595 20372 raft_consensus.cc:348] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.238693 20372 raft_consensus.cc:374] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:28.238744 20372 raft_consensus.cc:729] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Initialized, Role: FOLLOWER
I20260806 23:57:28.238852 20372 consensus_queue.cc:260] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.239084 20372 ts_tablet_manager.cc:1428] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:57:28.296986 20379 raft_consensus.cc:482] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:28.297160 20379 raft_consensus.cc:504] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.298211 20379 leader_election.cc:288] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587), 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911)
I20260806 23:57:28.303344 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.298555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:49227 (local address 127.8.204.66:32911)
0806 23:57:28.298682 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.298687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.298791 (+ 104us) server_negotiation.cc:407] Connection header received
0806 23:57:28.298923 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.298927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.298979 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.299065 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:28.299464 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.300178 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302031 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.302616 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302774 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.302820 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.303150 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.303215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:57:28.303344 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.298797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:46621 (local address 127.8.204.65:45587)
0806 23:57:28.298903 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.298908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.298921 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:28.298964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.298967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.299011 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.299083 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:28.299464 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.300178 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302031 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.302616 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302765 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.302820 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.303139 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.303224 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 23:57:28.303445 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.298431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45587 (local address 127.8.204.67:46621)
0806 23:57:28.298769 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:28.298780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:28.298838 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:28.299109 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:28.299112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:28.299150 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:28.299347 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.299353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.300307 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.300310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.301901 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.301914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.302788 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.302959 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.303025 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 23:57:28.303106 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 23:57:28.303236 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":417,"spinlock_wait_cycles":29184,"thread_start_us":61,"threads_started":1}
I20260806 23:57:28.303445 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.298431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.8.204.67:49227)
0806 23:57:28.298678 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:28.298693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:28.298824 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:28.299109 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:28.299112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:28.299152 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:28.299340 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.299348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.300307 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.300310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.301902 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.301914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.302788 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.302793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.302956 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.303022 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 23:57:28.303106 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 23:57:28.303236 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":227,"spinlock_wait_cycles":5632,"thread_start_us":72,"threads_started":1}
I20260806 23:57:28.304036 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bc812cc5d04ea3bb35d29bd29a2813" candidate_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971" is_pre_election: true
I20260806 23:57:28.304095 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bc812cc5d04ea3bb35d29bd29a2813" candidate_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bbff8ace5944f3841e2579e2210145" is_pre_election: true
I20260806 23:57:28.304179 20062 raft_consensus.cc:2454] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc85758b3bef4f0c82f62e3bdd8d44a3 in term 0.
I20260806 23:57:28.304296 19855 raft_consensus.cc:2454] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc85758b3bef4f0c82f62e3bdd8d44a3 in term 0.
I20260806 23:57:28.304601 20178 leader_election.cc:302] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, dc85758b3bef4f0c82f62e3bdd8d44a3; no voters:
I20260806 23:57:28.304852 20379 raft_consensus.cc:2781] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:28.304911 20379 raft_consensus.cc:482] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:28.304955 20379 raft_consensus.cc:3037] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.306051 20379 raft_consensus.cc:504] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.306330 20379 leader_election.cc:288] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [CANDIDATE]: Term 1 election: Requested vote from peers f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587), 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911)
I20260806 23:57:28.306648 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bc812cc5d04ea3bb35d29bd29a2813" candidate_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3bbff8ace5944f3841e2579e2210145"
I20260806 23:57:28.306694 20062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bc812cc5d04ea3bb35d29bd29a2813" candidate_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392a0e48b24b09ac002babde098971"
I20260806 23:57:28.306818 19855 raft_consensus.cc:3037] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.306849 20062 raft_consensus.cc:3037] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:28.307699 19855 raft_consensus.cc:2454] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc85758b3bef4f0c82f62e3bdd8d44a3 in term 1.
I20260806 23:57:28.307704 20062 raft_consensus.cc:2454] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc85758b3bef4f0c82f62e3bdd8d44a3 in term 1.
I20260806 23:57:28.307956 20178 leader_election.cc:302] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392a0e48b24b09ac002babde098971, dc85758b3bef4f0c82f62e3bdd8d44a3; no voters:
I20260806 23:57:28.308097 20379 raft_consensus.cc:2781] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:28.308324 20379 raft_consensus.cc:686] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 LEADER]: Becoming Leader. State: Replica: dc85758b3bef4f0c82f62e3bdd8d44a3, State: Running, Role: LEADER
I20260806 23:57:28.308561 20379 consensus_queue.cc:237] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } }
I20260806 23:57:28.310860 19622 catalog_manager.cc:5495] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67). New cstate: current_term: 1 leader_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc85758b3bef4f0c82f62e3bdd8d44a3" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41091 } health_report { overall_health: HEALTHY } } }
I20260806 23:57:28.340695 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.337435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.8.204.67:41091)
0806 23:57:28.337676 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.337683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.337698 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:28.337757 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.337759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.337803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.337895 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:28.338052 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.338522 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.339377 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.339578 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.339655 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.339877 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.340323 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.340403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.340469 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.340509 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
I20260806 23:57:28.350020 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.347348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:55229 (local address 127.8.204.66:32911)
0806 23:57:28.347484 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.347487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.347680 (+ 193us) server_negotiation.cc:407] Connection header received
0806 23:57:28.347766 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.347768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.347806 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.347859 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:28.348050 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.348445 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.349149 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.349436 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.349512 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.349727 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.349782 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.349831 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.349902 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.349937 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:57:28.350050 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.347288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:32911 (local address 127.8.204.67:55229)
0806 23:57:28.347614 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:28.347624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:28.347699 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:28.347884 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:28.347886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:28.347891 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:57:28.347973 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.347975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.348520 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.348522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.349060 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.349067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.349527 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.349529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.349589 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.349625 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.349849 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.349853 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:57:28.349895 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 23:57:28.349949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260806 23:57:28.355165 20062 raft_consensus.cc:1261] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Refusing update from remote peer dc85758b3bef4f0c82f62e3bdd8d44a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:28.355242 19855 raft_consensus.cc:1261] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Refusing update from remote peer dc85758b3bef4f0c82f62e3bdd8d44a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:28.355608 20489 consensus_queue.cc:1035] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392a0e48b24b09ac002babde098971" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:28.355800 20379 consensus_queue.cc:1035] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3bbff8ace5944f3841e2579e2210145" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:28.371824 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.369114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:42049 (local address 127.8.204.67:41091)
0806 23:57:28.369238 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:28.369242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:28.369254 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:28.369342 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:28.369344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:28.369390 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:28.369468 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:28.369655 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.370061 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.370812 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.371003 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.371074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.371378 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.371463 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.371535 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:28.371621 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 23:57:28.371689 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:57:28.371840 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.368904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41091 (local address 127.8.204.66:42049)
0806 23:57:28.369163 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:28.369178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:28.369257 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:28.369489 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:28.369491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:28.369498 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:57:28.369572 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.369576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.370145 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.370147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.370659 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:28.370666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:28.371086 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:28.371090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:28.371167 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:28.371209 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:57:28.371546 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:57:28.371551 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:57:28.371600 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 23:57:28.371715 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260806 23:57:28.393435 19620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:28.393612 19620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:28.395608 19620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 964f224a465249c7b34422b0e2c2d812: Sending DeleteTablet for 3 replicas of tablet b3bc812cc5d04ea3bb35d29bd29a2813
I20260806 23:57:28.396154 19828 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53478
I20260806 23:57:28.396345 20407 tablet_replica.cc:330] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145: stopping tablet replica
I20260806 23:57:28.396476 20407 raft_consensus.cc:2229] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:28.396579 20407 raft_consensus.cc:2258] T b3bc812cc5d04ea3bb35d29bd29a2813 P f3bbff8ace5944f3841e2579e2210145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.396632 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35656
I20260806 23:57:28.396798 20035 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bc812cc5d04ea3bb35d29bd29a2813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:28 UTC) from {username='oryx', principal='oryx/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44978
I20260806 23:57:28.396827 20406 tablet_replica.cc:330] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: stopping tablet replica
I20260806 23:57:28.396943 20408 tablet_replica.cc:330] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: stopping tablet replica
I20260806 23:57:28.396948 20406 raft_consensus.cc:2229] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:57:28.397063 20408 raft_consensus.cc:2229] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:28.397065 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 19727
I20260806 23:57:28.397151 20408 raft_consensus.cc:2258] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.397222 20406 raft_consensus.cc:2258] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:28.403545 20408 ts_tablet_manager.cc:1905] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:28.404008 20406 ts_tablet_manager.cc:1905] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:28.406020 20408 ts_tablet_manager.cc:1918] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:28.406109 20408 log.cc:1196] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/b3bc812cc5d04ea3bb35d29bd29a2813
I20260806 23:57:28.406383 20408 ts_tablet_manager.cc:1939] T b3bc812cc5d04ea3bb35d29bd29a2813 P 3d392a0e48b24b09ac002babde098971: Deleting consensus metadata
I20260806 23:57:28.406980 20406 ts_tablet_manager.cc:1918] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:57:28.407064 20406 log.cc:1196] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/b3bc812cc5d04ea3bb35d29bd29a2813
I20260806 23:57:28.407364 20406 ts_tablet_manager.cc:1939] T b3bc812cc5d04ea3bb35d29bd29a2813 P dc85758b3bef4f0c82f62e3bdd8d44a3: Deleting consensus metadata
I20260806 23:57:28.407498 19599 catalog_manager.cc:4841] TS 3d392a0e48b24b09ac002babde098971 (127.8.204.66:32911): tablet b3bc812cc5d04ea3bb35d29bd29a2813 (table test-table [id=861db7ec4d674aaaa2924897c4d89d62]) successfully deleted
I20260806 23:57:28.408046 19601 catalog_manager.cc:4841] TS dc85758b3bef4f0c82f62e3bdd8d44a3 (127.8.204.67:41091): tablet b3bc812cc5d04ea3bb35d29bd29a2813 (table test-table [id=861db7ec4d674aaaa2924897c4d89d62]) successfully deleted
I20260806 23:57:28.413708 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 19932
W20260806 23:57:28.413688 19600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:45587 (error 108)
I20260806 23:57:28.426023 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:28.425478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45587 (local address 127.0.0.1:53516)
0806 23:57:28.425820 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:28.425903 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":61,"threads_started":1}
W20260806 23:57:28.426141 19600 catalog_manager.cc:4584] TS f3bbff8ace5944f3841e2579e2210145 (127.8.204.65:45587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3bc812cc5d04ea3bb35d29bd29a2813: Network error: Client connection negotiation failed: client connection to 127.8.204.65:45587: connect: Connection refused (error 111)
I20260806 23:57:28.427366 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 20149
I20260806 23:57:28.439751 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 19574
2026-08-06T23:57:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3388 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:28 dist-test-slave-39l8 krb5kdc[20510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:28 dist-test-slave-39l8 krb5kdc[20510](info): set up 2 sockets
Aug 06 23:57:28 dist-test-slave-39l8 krb5kdc[20510](info): commencing operation
krb5kdc: starting...
W20260806 23:57:29.582139 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:29 dist-test-slave-39l8 krb5kdc[20510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:29.695619 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37417
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:29.801576 20526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:29.801848 20526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:29.801915 20526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:29.805639 20526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:29.805717 20526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:29.805753 20526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:29.805805 20526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:29.805841 20526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:29.810601 20526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:29.811544 20526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:29.812500 20526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:29.820366 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:29.820458 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:29.820379 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:29.820607 20526 server_base.cc:1034] running on GCE node
I20260806 23:57:29.821190 20526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:29.822161 20526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:29.823339 20526 hybrid_clock.cc:648] HybridClock initialized: now 1786060649823313 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T23:57:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1388 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:29 dist-test-slave-39l8 krb5kdc[20545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:29 dist-test-slave-39l8 krb5kdc[20545](info): set up 2 sockets
Aug 06 23:57:29 dist-test-slave-39l8 krb5kdc[20545](info): commencing operation
krb5kdc: starting...
W20260806 23:57:31.006807 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:31.129735 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40697
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:31.239708 20561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:31.239982 20561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:31.240061 20561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:31.244035 20561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:31.244112 20561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:31.244156 20561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:31.244237 20561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:31.244272 20561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:31.249368 20561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:31.250319 20561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:31.251338 20561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:31.259279 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.259270 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.259451 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:31.259307 20561 server_base.cc:1034] running on GCE node
I20260806 23:57:31.260030 20561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:31.261006 20561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:31.262202 20561 hybrid_clock.cc:648] HybridClock initialized: now 1786060651262180 us; error 41 us; skew 500 ppm
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:31.264899 20561 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:31.265666 20561 webserver.cc:460] Webserver started at http://127.8.204.126:33599/ using document root <none> and password file <none>
I20260806 23:57:31.266161 20561 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:31.266273 20561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:31.266481 20561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:31.268374 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "e57b9c348c5845618394a612c6a68f89"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "ba03ea5a2eec997688dc70932b819cd3"
server_key_iv: "14dc5447e0af0dc286c4902639305cf4"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.268899 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "e57b9c348c5845618394a612c6a68f89"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "ba03ea5a2eec997688dc70932b819cd3"
server_key_iv: "14dc5447e0af0dc286c4902639305cf4"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.272464 20561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:31.274626 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.275774 20561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:31.275909 20561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "e57b9c348c5845618394a612c6a68f89"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "ba03ea5a2eec997688dc70932b819cd3"
server_key_iv: "14dc5447e0af0dc286c4902639305cf4"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.276013 20561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:31.288455 20561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:31.291465 20561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:31.291657 20561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:31.298504 20561 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40697
I20260806 23:57:31.298569 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40697 every 8 connection(s)
I20260806 23:57:31.299410 20561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:31.302059 20667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:31.306456 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 20561
I20260806 23:57:31.306618 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:31.306955 9009 external_mini_cluster.cc:1428] Setting key 9029c07004c6b35ca2f65ab901abb6f9
I20260806 23:57:31.307703 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Bootstrap starting.
I20260806 23:57:31.309926 20667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:31.310874 20667 log.cc:824] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:31.312918 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: No bootstrap required, opened a new log
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:31.317009 20667 raft_consensus.cc:348] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:31.317210 20667 raft_consensus.cc:374] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:31.317278 20667 raft_consensus.cc:729] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57b9c348c5845618394a612c6a68f89, State: Initialized, Role: FOLLOWER
I20260806 23:57:31.317734 20667 consensus_queue.cc:260] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:31.317849 20667 raft_consensus.cc:388] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:31.317945 20667 raft_consensus.cc:482] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:31.318035 20667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:31.318913 20667 raft_consensus.cc:504] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:31.319247 20667 leader_election.cc:302] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e57b9c348c5845618394a612c6a68f89; no voters:
I20260806 23:57:31.319552 20667 leader_election.cc:288] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:31.319672 20675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:31.319900 20675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 LEADER]: Becoming Leader. State: Replica: e57b9c348c5845618394a612c6a68f89, State: Running, Role: LEADER
I20260806 23:57:31.320281 20675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:31.320804 20667 sys_catalog.cc:564] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:31.321305 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:31.308225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.8.204.126:40697)
0806 23:57:31.308610 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:31.308625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:31.308657 (+ 32us) server_negotiation.cc:407] Connection header received
0806 23:57:31.309230 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:31.309245 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:31.309524 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:31.309779 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:31.310646 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.311628 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.312803 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.313285 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.313398 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:31.316082 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:31.316087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:31.316098 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:31.316124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:31.317968 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.318412 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.318418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.318423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.318535 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.318860 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.318863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.318865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.319272 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:31.319501 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:31.320117 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 23:57:31.320392 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260806 23:57:31.321806 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e57b9c348c5845618394a612c6a68f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } } }
I20260806 23:57:31.321933 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:31.322316 20685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:31.322299 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e57b9c348c5845618394a612c6a68f89. Latest consensus state: current_term: 1 leader_uuid: "e57b9c348c5845618394a612c6a68f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } } }
I20260806 23:57:31.322453 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:31.325366 20685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:31.330299 20685 catalog_manager.cc:1347] Generated new cluster ID: 4f3a7bc3da034250b99472cf98ba5049
I20260806 23:57:31.330363 20685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:31.341346 20685 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:31.342460 20685 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:31.348249 20685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Generated new TSK 0
I20260806 23:57:31.348887 20685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:31.381213 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:31.491089 20705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:31.491309 20705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:31.491348 20705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:31.495322 20705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:31.495373 20705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:31.495425 20705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:31.500350 20705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:31.501228 20705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:31.502135 20705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:31.511096 20711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.511132 20714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.511139 20710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:31.511297 20705 server_base.cc:1034] running on GCE node
I20260806 23:57:31.511688 20705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:31.512301 20705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:31.513445 20705 hybrid_clock.cc:648] HybridClock initialized: now 1786060651513425 us; error 31 us; skew 500 ppm
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:31.520582 20705 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:31.521454 20705 webserver.cc:460] Webserver started at http://127.8.204.65:39351/ using document root <none> and password file <none>
I20260806 23:57:31.522032 20705 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:31.522117 20705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:31.522368 20705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:31.524485 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "f4b154a0ce804754bc2579110ba13df0"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "c689a1599849993f209516e54f195c9d"
server_key_iv: "99dcf5ca112974d2c4e540ec1266954f"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.525025 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "f4b154a0ce804754bc2579110ba13df0"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "c689a1599849993f209516e54f195c9d"
server_key_iv: "99dcf5ca112974d2c4e540ec1266954f"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.528565 20705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:31.530954 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.532114 20705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:31.532258 20705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "f4b154a0ce804754bc2579110ba13df0"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "c689a1599849993f209516e54f195c9d"
server_key_iv: "99dcf5ca112974d2c4e540ec1266954f"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.532369 20705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:31.563563 20705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:31.566843 20705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:31.567039 20705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:31.567625 20705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:31.568436 20705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:31.568596 20705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.568722 20705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:31.568774 20705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.577797 20705 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:42179
I20260806 23:57:31.577832 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:42179 every 8 connection(s)
I20260806 23:57:31.578684 20705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:31.587638 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 20705
I20260806 23:57:31.587766 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:31.588053 9009 external_mini_cluster.cc:1428] Setting key eca38b73b263b3150abf3ccf653376b7
I20260806 23:57:31.592170 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:31.580534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:60027 (local address 127.8.204.126:40697)
0806 23:57:31.580667 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:31.580672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:31.581443 (+ 771us) server_negotiation.cc:407] Connection header received
0806 23:57:31.582677 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:31.582680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:31.582756 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:31.582831 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:31.583911 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.584415 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.585513 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.585735 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.585812 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:31.588357 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:31.588361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:31.588363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:31.588390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:31.590073 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.590550 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.590556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.590561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.590663 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.591027 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.591032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.591036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.591289 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:31.591396 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:31.591856 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 23:57:31.592000 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:31.592952 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:31.580848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:60027)
0806 23:57:31.581284 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:31.581326 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:31.582447 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:31.582968 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:31.582974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:31.583303 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:31.583760 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:31.583773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.584538 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.584540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:31.585399 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:31.585409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.585864 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.585867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:31.586000 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:31.586587 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:31.586602 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:31.588184 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:31.590191 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:31.590196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:31.590206 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:31.590437 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:31.590811 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:31.590813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:31.590814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:31.590933 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:31.591397 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:31.591401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:31.591611 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 23:57:31.591801 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 23:57:31.592117 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260806 23:57:31.594197 20876 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40697
I20260806 23:57:31.594456 20876 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:31.595094 20876 heartbeater.cc:511] Master 127.8.204.126:40697 requested a full tablet report, sending...
I20260806 23:57:31.596783 20608 ts_manager.cc:194] Registered new tserver with Master: f4b154a0ce804754bc2579110ba13df0 (127.8.204.65:42179)
I20260806 23:57:31.598385 20608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:60027
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:31.617045 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:31.724224 20885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:31.724431 20885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:31.724496 20885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:31.728149 20885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:31.728195 20885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:31.728263 20885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:31.733013 20885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:31.733950 20885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:31.734848 20885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:31.743909 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.743923 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:31.743979 20885 server_base.cc:1034] running on GCE node
W20260806 23:57:31.743916 20894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:31.744563 20885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:31.745167 20885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:31.746366 20885 hybrid_clock.cc:648] HybridClock initialized: now 1786060651746352 us; error 34 us; skew 500 ppm
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:31.749075 20885 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:31.749943 20885 webserver.cc:460] Webserver started at http://127.8.204.66:35719/ using document root <none> and password file <none>
I20260806 23:57:31.750520 20885 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:31.750618 20885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:31.750874 20885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:31.752909 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "4315a34bd2314b798d1c25c55f75c114"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "b397e388d30927178378bdd93f1c4407"
server_key_iv: "e967de36efdf3232e334a08a37e65758"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.753469 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "4315a34bd2314b798d1c25c55f75c114"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "b397e388d30927178378bdd93f1c4407"
server_key_iv: "e967de36efdf3232e334a08a37e65758"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.757266 20885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:57:31.759519 20902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.760589 20885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 23:57:31.760730 20885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "4315a34bd2314b798d1c25c55f75c114"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "b397e388d30927178378bdd93f1c4407"
server_key_iv: "e967de36efdf3232e334a08a37e65758"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.760831 20885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:31.771513 20885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:31.774276 20885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:31.774457 20885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:31.775038 20885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:31.775857 20885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:31.775926 20885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.775991 20885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:31.776038 20885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.784596 20885 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:40847
I20260806 23:57:31.784621 21053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:40847 every 8 connection(s)
I20260806 23:57:31.785362 20885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:31.793254 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 20885
I20260806 23:57:31.793407 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:31.793718 9009 external_mini_cluster.cc:1428] Setting key 99bdc9a2f9230d3da95297f315366e2d
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:31.797803 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:31.787024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48385 (local address 127.8.204.126:40697)
0806 23:57:31.787149 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:31.787153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:31.788000 (+ 847us) server_negotiation.cc:407] Connection header received
0806 23:57:31.788796 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:31.788799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:31.788856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:31.788956 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:31.789958 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.790441 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.791504 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.791711 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.791815 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:31.794344 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:31.794348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:31.794350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:31.794377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:31.795884 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.796345 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.796349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.796350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.796432 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:31.796791 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:31.796794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:31.796796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:31.796966 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:31.797073 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:31.797522 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 23:57:31.797631 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:57:31.798482 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:31.787301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:48385)
0806 23:57:31.787866 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:31.787902 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:31.788598 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:31.789119 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:31.789125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:31.789424 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:31.789824 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:31.789835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.790560 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.790568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:31.791381 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:31.791390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:31.791850 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:31.791855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:31.792010 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:31.792580 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:31.792596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:31.794161 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:31.795998 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:31.796004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:31.796015 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:31.796247 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:31.796547 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:31.796550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:31.796552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:31.796691 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:31.797073 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:31.797081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:31.797293 (+ 212us) client_negotiation.cc:769] Sending connection context
0806 23:57:31.797465 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 23:57:31.797769 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":95,"threads_started":1}
I20260806 23:57:31.799607 21054 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40697
I20260806 23:57:31.799860 21054 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:31.800284 21054 heartbeater.cc:511] Master 127.8.204.126:40697 requested a full tablet report, sending...
I20260806 23:57:31.801249 20608 ts_manager.cc:194] Registered new tserver with Master: 4315a34bd2314b798d1c25c55f75c114 (127.8.204.66:40847)
I20260806 23:57:31.801983 20608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:48385
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:31.821214 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:31.930444 21061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:31.930670 21061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:31.930766 21061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:31.934602 21061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:31.934649 21061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:31.934744 21061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:31.939460 21061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:31.940317 21061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:31.941226 21061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:31.950177 21072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.950177 21069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:31.950204 21068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:31.950415 21061 server_base.cc:1034] running on GCE node
I20260806 23:57:31.950871 21061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:31.951433 21061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:31.952623 21061 hybrid_clock.cc:648] HybridClock initialized: now 1786060651952612 us; error 30 us; skew 500 ppm
Aug 06 23:57:31 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:31.955351 21061 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:31.956234 21061 webserver.cc:460] Webserver started at http://127.8.204.67:35971/ using document root <none> and password file <none>
I20260806 23:57:31.956744 21061 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:31.956830 21061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:31.957051 21061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:31.958875 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "59648a12ad8045609d5b0b9695a9d14a"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "89cbcde120d38a3ed5648c9eccadddc6"
server_key_iv: "13d7b9abbc79731933f06dfe14cd686c"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.959400 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "59648a12ad8045609d5b0b9695a9d14a"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "89cbcde120d38a3ed5648c9eccadddc6"
server_key_iv: "13d7b9abbc79731933f06dfe14cd686c"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.962798 21061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:57:31.965126 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.966238 21061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 23:57:31.966373 21061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "59648a12ad8045609d5b0b9695a9d14a"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "89cbcde120d38a3ed5648c9eccadddc6"
server_key_iv: "13d7b9abbc79731933f06dfe14cd686c"
server_key_version: "encryptionkey@0"
I20260806 23:57:31.966476 21061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:31.994340 21061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:31.997161 21061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:31.997339 21061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:31.997925 21061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:31.998668 21061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:31.998755 21061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:31.998827 21061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:31.998876 21061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.007553 21061 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:44253
I20260806 23:57:32.007587 21250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:44253 every 8 connection(s)
I20260806 23:57:32.008345 21061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:32.017609 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21061
I20260806 23:57:32.017748 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:32.018863 9009 external_mini_cluster.cc:1428] Setting key a3e1e7cb0af9a014ff4ea6b4e687f7ec
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060651, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:32.023959 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.010033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:51767 (local address 127.8.204.126:40697)
0806 23:57:32.010194 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.010198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.011164 (+ 966us) server_negotiation.cc:407] Connection header received
0806 23:57:32.012115 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.012118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.012197 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.012298 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.013936 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.014964 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.016619 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.016834 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.016946 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.020046 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:32.020049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:32.020052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:32.020106 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:32.021830 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:32.022331 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:32.022335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:32.022337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:32.022420 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:32.022886 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:32.022889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:32.022891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:32.023076 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:32.023188 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:32.023666 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 23:57:32.023790 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:57:32.024648 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.010401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:51767)
0806 23:57:32.011054 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.011090 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.011853 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.012510 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.012521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.013056 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.013748 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.013769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.015107 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.015113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.016490 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.016505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.016985 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.016990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.017197 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.017830 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.017848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.019882 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:32.021949 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:32.021954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:32.021967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:32.022218 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:32.022554 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:32.022557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:32.022559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:32.022776 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:32.023194 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:32.023203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:32.023429 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 23:57:32.023607 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:57:32.023911 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":105,"threads_started":1}
I20260806 23:57:32.025666 21253 heartbeater.cc:347] Connected to a master server at 127.8.204.126:40697
I20260806 23:57:32.025862 21253 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:32.026261 21253 heartbeater.cc:511] Master 127.8.204.126:40697 requested a full tablet report, sending...
I20260806 23:57:32.027191 20608 ts_manager.cc:194] Registered new tserver with Master: 59648a12ad8045609d5b0b9695a9d14a (127.8.204.67:44253)
I20260806 23:57:32.027673 20608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:51767
I20260806 23:57:32.032943 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
Entry for principal oryx/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab.
WARNING: no policy specified for oryx/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
Entry for principal oryx/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab.
WARNING: no policy specified for oryx/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
Entry for principal oryx/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab.
I20260806 23:57:32.098237 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 20705
I20260806 23:57:32.108068 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 20885
I20260806 23:57:32.117749 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21061
I20260806 23:57:32.128158 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 20561
I20260806 23:57:32.140120 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40697
--webserver_interface=127.8.204.126
--webserver_port=33599
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:32.246296 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:32.246537 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:32.246611 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:32.250334 21267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:32.250408 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:32.250444 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:32.250478 21267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:32.250505 21267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:32.255370 21267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:32.256244 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:32.257135 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:32.264724 21267 server_base.cc:1034] running on GCE node
W20260806 23:57:32.264616 21276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.264634 21272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.264658 21273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:32.265331 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:32.266295 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:32.269557 21267 hybrid_clock.cc:648] HybridClock initialized: now 1786060652269534 us; error 30 us; skew 500 ppm
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060652, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:32.272214 21267 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:32.273092 21267 webserver.cc:460] Webserver started at http://127.8.204.126:33599/ using document root <none> and password file <none>
I20260806 23:57:32.273618 21267 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:32.273723 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:32.277199 21267 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:32.279080 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.280134 21267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:32.280265 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "e57b9c348c5845618394a612c6a68f89"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "ba03ea5a2eec997688dc70932b819cd3"
server_key_iv: "14dc5447e0af0dc286c4902639305cf4"
server_key_version: "encryptionkey@0"
I20260806 23:57:32.280707 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:32.291622 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:32.294613 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:32.294838 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:32.301252 21267 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40697
I20260806 23:57:32.301261 21354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40697 every 8 connection(s)
I20260806 23:57:32.302103 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:32.305882 21355 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:57:32.305917 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21267
I20260806 23:57:32.306394 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:42179
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:40697
--webserver_port=39351
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:57:32.307873 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Bootstrap starting.
I20260806 23:57:32.323544 21355 log.cc:824] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:32.329524 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:57:32.329874 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Bootstrap complete.
I20260806 23:57:32.334062 21355 raft_consensus.cc:348] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:32.334498 21355 raft_consensus.cc:729] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57b9c348c5845618394a612c6a68f89, State: Initialized, Role: FOLLOWER
I20260806 23:57:32.335042 21355 consensus_queue.cc:260] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:32.335152 21355 raft_consensus.cc:388] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:32.335232 21355 raft_consensus.cc:482] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:32.335355 21355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:57:32.337003 21355 raft_consensus.cc:504] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:32.337309 21355 leader_election.cc:302] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e57b9c348c5845618394a612c6a68f89; no voters:
I20260806 23:57:32.337596 21355 leader_election.cc:288] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:57:32.337714 21361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:57:32.337930 21361 raft_consensus.cc:686] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [term 2 LEADER]: Becoming Leader. State: Replica: e57b9c348c5845618394a612c6a68f89, State: Running, Role: LEADER
I20260806 23:57:32.338199 21361 consensus_queue.cc:237] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } }
I20260806 23:57:32.338649 21355 sys_catalog.cc:564] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:32.339071 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e57b9c348c5845618394a612c6a68f89. Latest consensus state: current_term: 2 leader_uuid: "e57b9c348c5845618394a612c6a68f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } } }
I20260806 23:57:32.339197 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:32.339805 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e57b9c348c5845618394a612c6a68f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57b9c348c5845618394a612c6a68f89" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40697 } } }
I20260806 23:57:32.339902 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:32.340322 21365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:32.343259 21365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:32.343918 21365 catalog_manager.cc:1259] Loaded cluster ID: 4f3a7bc3da034250b99472cf98ba5049
I20260806 23:57:32.344000 21365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:32.346216 21365 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:32.346825 21365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e57b9c348c5845618394a612c6a68f89: Loaded TSK: 0
I20260806 23:57:32.347899 21365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:57:32.414490 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:32.414675 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:32.414760 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:32.418265 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:32.418309 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:32.418357 21359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:32.422897 21359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.65.keytab
--principal=oryx/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:42179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=39351
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:32.423765 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:32.424641 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:32.433596 21389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.433749 21390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.433602 21392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:32.434297 21359 server_base.cc:1034] running on GCE node
I20260806 23:57:32.434680 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:32.435400 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:32.436573 21359 hybrid_clock.cc:648] HybridClock initialized: now 1786060652436562 us; error 33 us; skew 500 ppm
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060652, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:32.439386 21359 init.cc:377] Logged in from keytab as oryx/127.8.204.65@KRBTEST.COM (short username oryx)
I20260806 23:57:32.440308 21359 webserver.cc:460] Webserver started at http://127.8.204.65:39351/ using document root <none> and password file <none>
I20260806 23:57:32.440894 21359 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:32.441005 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:32.444710 21359 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 23:57:32.446641 21403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.447798 21359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 23:57:32.447924 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "f4b154a0ce804754bc2579110ba13df0"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "c689a1599849993f209516e54f195c9d"
server_key_iv: "99dcf5ca112974d2c4e540ec1266954f"
server_key_version: "encryptionkey@0"
I20260806 23:57:32.448378 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:32.464792 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:32.467825 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:32.468025 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:32.468554 21359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:32.469317 21359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:32.469385 21359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.469451 21359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:32.469501 21359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.477910 21359 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:42179
I20260806 23:57:32.477937 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:42179 every 8 connection(s)
I20260806 23:57:32.478857 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:32.482963 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21359
I20260806 23:57:32.483443 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:40847
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:40697
--webserver_port=35719
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.491575 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.480694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:35977)
0806 23:57:32.481262 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.481304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.482001 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.482974 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.482979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.483343 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.483857 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.483870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.485555 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.485560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.486872 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.486885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.487852 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.487855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.487979 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.488521 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.488537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.490976 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":118,"threads_started":1}
W20260806 23:57:32.491993 21554 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:40697 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.126:40697: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
W20260806 23:57:32.492087 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.480642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35977 (local address 127.8.204.126:40697)
0806 23:57:32.481010 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.481023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.481400 (+ 377us) server_negotiation.cc:407] Connection header received
0806 23:57:32.482298 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.482307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.482546 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.482941 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.484032 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.485423 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.486993 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.487695 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.487875 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.491254 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:35977: BlockingRecv error: recv got EOF from 127.8.204.65:35977 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.497669 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.492418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:50811)
0806 23:57:32.492604 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.492617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.492691 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.492968 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.492970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.493096 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.493180 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.493184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.494233 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.494235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.494688 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.494695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.495322 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.495325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.495385 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.495790 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.495800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.497576 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":101}
W20260806 23:57:32.497822 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.492578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:50811 (local address 127.8.204.126:40697)
0806 23:57:32.492713 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.492719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.492753 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:57:32.492794 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.492799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.492856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.492956 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.493294 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.494120 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.494839 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.495218 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.495337 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.497720 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:50811: BlockingRecv error: recv got EOF from 127.8.204.65:50811 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.502476 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.498101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:32883)
0806 23:57:32.498228 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.498258 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.498336 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.498581 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.498584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.498771 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.498844 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.498849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.499469 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.499473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.499900 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.499907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.500295 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.500299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.500366 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.500777 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.500789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.502363 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 23:57:32.502554 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.498213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:32883 (local address 127.8.204.126:40697)
0806 23:57:32.498332 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.498336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.498349 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:32.498426 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.498428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.498471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.498567 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.498946 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.499371 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.500015 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.500193 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.500284 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.502461 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:32883: BlockingRecv error: recv got EOF from 127.8.204.65:32883 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 23:57:32.502681 21554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:57:32.593825 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:32.594074 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:32.594111 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:32.597781 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:32.597831 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:32.597908 21559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:32.602923 21559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.66.keytab
--principal=oryx/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:40847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=35719
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:32.603896 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:32.604813 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:32.613500 21570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.613497 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.613686 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:32.613951 21559 server_base.cc:1034] running on GCE node
I20260806 23:57:32.614377 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:32.615046 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:32.616226 21559 hybrid_clock.cc:648] HybridClock initialized: now 1786060652616203 us; error 44 us; skew 500 ppm
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060652, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:32.618793 21559 init.cc:377] Logged in from keytab as oryx/127.8.204.66@KRBTEST.COM (short username oryx)
I20260806 23:57:32.619606 21559 webserver.cc:460] Webserver started at http://127.8.204.66:35719/ using document root <none> and password file <none>
I20260806 23:57:32.620095 21559 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:32.620165 21559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:32.623690 21559 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:32.625484 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.626538 21559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:32.626644 21559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "4315a34bd2314b798d1c25c55f75c114"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "b397e388d30927178378bdd93f1c4407"
server_key_iv: "e967de36efdf3232e334a08a37e65758"
server_key_version: "encryptionkey@0"
I20260806 23:57:32.627166 21559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:32.645607 21559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:32.648039 21559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:32.648206 21559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:32.648722 21559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:32.649441 21559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:32.649484 21559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.649549 21559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:32.649607 21559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.658653 21559 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:40847
I20260806 23:57:32.658674 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:40847 every 8 connection(s)
I20260806 23:57:32.659538 21559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:32.661032 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21559
I20260806 23:57:32.661510 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:44253
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:40697
--webserver_port=35971
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:57:32.671118 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.661174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:34489 (local address 127.8.204.126:40697)
0806 23:57:32.661328 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.661334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.662429 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 23:57:32.663768 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.663774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.663842 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.663964 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.665055 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.665526 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.667281 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.667479 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.667574 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.671003 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:34489: BlockingRecv error: recv got EOF from 127.8.204.66:34489 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:57:32.671643 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.661503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:34489)
0806 23:57:32.662272 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.662323 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.663543 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.664104 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.664109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.664436 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.664894 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.664915 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.665654 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.665660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.667102 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.667160 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.667645 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.667651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.667868 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.668423 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.668448 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.670898 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":119,"threads_started":1}
W20260806 23:57:32.672197 21740 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:40697 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.126:40697: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.677868 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.672592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:57521)
0806 23:57:32.672698 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.672714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.672785 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.673181 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.673183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.673302 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.673387 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.673391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.674451 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.674454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.675030 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.675040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.675570 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.675572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.675631 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.676091 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.676101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.677767 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 23:57:32.677989 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.672787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:57521 (local address 127.8.204.126:40697)
0806 23:57:32.672930 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.672937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.672953 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:57:32.673000 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.673005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.673062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.673171 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.673501 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.674339 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.675142 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.675471 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.675564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.677870 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:57521: BlockingRecv error: recv got EOF from 127.8.204.66:57521 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.683358 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.678416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:55565)
0806 23:57:32.678592 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.678602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.678654 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.678862 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.678864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.678963 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.679042 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.679046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.679967 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.679970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.680456 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.680464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.680971 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.680973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.681025 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.681413 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.681423 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.683242 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260806 23:57:32.683533 21740 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:57:32.683560 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.678516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:55565 (local address 127.8.204.126:40697)
0806 23:57:32.678620 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.678624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.678636 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:32.678734 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.678736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.678770 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.678838 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.679143 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.679849 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.680555 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.680873 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.680964 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.683351 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:55565: BlockingRecv error: recv got EOF from 127.8.204.66:55565 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 23:57:32.768837 21745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:32.769039 21745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:32.769078 21745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:32.772783 21745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:32.772835 21745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:32.772938 21745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:32.777745 21745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/krb5kdc/oryx_127.8.204.67.keytab
--principal=oryx/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:44253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=35971
--tserver_master_addrs=127.8.204.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:32.778684 21745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:32.779635 21745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:32.788542 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:32.788594 21745 server_base.cc:1034] running on GCE node
W20260806 23:57:32.788549 21751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:32.788549 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:32.789170 21745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:32.789713 21745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:32.790884 21745 hybrid_clock.cc:648] HybridClock initialized: now 1786060652790858 us; error 40 us; skew 500 ppm
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060652, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:32.793372 21745 init.cc:377] Logged in from keytab as oryx/127.8.204.67@KRBTEST.COM (short username oryx)
I20260806 23:57:32.794211 21745 webserver.cc:460] Webserver started at http://127.8.204.67:35971/ using document root <none> and password file <none>
I20260806 23:57:32.794745 21745 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:32.794844 21745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:32.798250 21745 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:32.800115 21760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.801155 21745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:32.801287 21745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "59648a12ad8045609d5b0b9695a9d14a"
format_stamp: "Formatted at 2026-08-06 23:57:31 on dist-test-slave-39l8"
server_key: "89cbcde120d38a3ed5648c9eccadddc6"
server_key_iv: "13d7b9abbc79731933f06dfe14cd686c"
server_key_version: "encryptionkey@0"
I20260806 23:57:32.801735 21745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:32.845494 21745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:32.848104 21745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:32.848292 21745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:32.848923 21745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:32.849663 21745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:32.849730 21745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.849795 21745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:32.849843 21745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:32.858594 21745 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:44253
I20260806 23:57:32.858619 21907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:44253 every 8 connection(s)
I20260806 23:57:32.859486 21745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.871201 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.861415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:34815)
0806 23:57:32.862055 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.862089 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.862912 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.863443 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.863449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.863768 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.864210 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.864222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.864951 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.864954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.865957 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.865968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.866669 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.866675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.866950 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.867678 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.867698 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.870337 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":97,"threads_started":1}
I20260806 23:57:32.873634 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21745
W20260806 23:57:32.873776 21908 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:40697 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.126:40697: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
W20260806 23:57:32.874830 21557 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.861081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:34815 (local address 127.8.204.126:40697)
0806 23:57:32.861202 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.861206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.862178 (+ 972us) server_negotiation.cc:407] Connection header received
0806 23:57:32.863109 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.863113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.863171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.863277 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.864346 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.864824 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.866097 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.866492 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.866603 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.874651 (+ 8048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:34815: BlockingRecv error: recv got EOF from 127.8.204.67:34815 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.879658 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.874209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:54861)
0806 23:57:32.874338 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.874350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.874408 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.875098 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.875100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.875248 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.875346 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.875352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.876184 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.876189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.876867 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.876876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.877373 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.877376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.877458 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.877822 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.877835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.879545 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 23:57:32.879755 21916 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.874583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:54861 (local address 127.8.204.126:40697)
0806 23:57:32.874836 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.874850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.874864 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:32.874940 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.874943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.874987 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.875078 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.875458 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.876081 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.876963 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.877274 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.877357 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.879642 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:54861: BlockingRecv error: recv got EOF from 127.8.204.67:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260806 23:57:32.884160 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.874612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.8.204.126:40697)
0806 23:57:32.874979 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.874982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.875708 (+ 726us) server_negotiation.cc:407] Connection header received
0806 23:57:32.875798 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.875800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.875837 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.875901 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.876654 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.877187 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.878333 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.878658 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.878781 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.880200 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:32.880208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:32.880224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:32.880273 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:32.882050 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:32.882482 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:32.882487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:32.882492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:32.882612 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:32.882972 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:32.882976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:32.882978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:32.883279 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:32.883478 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:32.883753 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 23:57:32.884009 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":85,"threads_started":1}
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:32 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:32.885324 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:32.880113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:49093)
0806 23:57:32.880227 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:32.880239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:32.880322 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:32.880585 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:32.880589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:32.880749 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:32.880835 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.880843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.881798 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.881800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.882371 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:32.882380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.882928 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.882935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:32.883019 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.883451 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:32.883465 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:32.885253 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 23:57:32.885469 21916 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:32.880183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:49093 (local address 127.8.204.126:40697)
0806 23:57:32.880355 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:32.880359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:32.880371 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:32.880404 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:32.880409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:32.880449 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:32.880537 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:32.880959 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.881691 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.882473 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:32.882815 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:32.882895 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:32.885381 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:49093: BlockingRecv error: recv got EOF from 127.8.204.67:49093 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260806 23:57:32.885871 21908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:33.508687 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:33.503295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:57759)
0806 23:57:33.503572 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:33.503589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:33.503708 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:33.503991 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:33.503994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:33.504195 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:33.504379 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.504386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.505119 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.505123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.505731 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.505741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.506208 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.506212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.506315 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.506781 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:33.506792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:33.508540 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260806 23:57:33.508777 21919 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:33.503333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:57759 (local address 127.8.204.126:40697)
0806 23:57:33.503597 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:33.503603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:33.503653 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:57:33.503815 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:33.503819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:33.503873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:33.503969 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:33.504492 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.505013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.505850 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.506073 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.506182 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.508650 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:57759: BlockingRecv error: recv got EOF from 127.8.204.65:57759 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:33.689332 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:33.684099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:33779)
0806 23:57:33.684347 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:33.684359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:33.684449 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:33.684715 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:33.684718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:33.684918 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:33.685052 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.685062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.685898 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.685903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.686416 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.686424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.686943 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.686946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.687027 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.687537 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:33.687551 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:33.689224 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
W20260806 23:57:33.689489 21919 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:33.684187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:33779 (local address 127.8.204.126:40697)
0806 23:57:33.684329 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:33.684333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:33.684409 (+ 76us) server_negotiation.cc:407] Connection header received
0806 23:57:33.684511 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:33.684513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:33.684582 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:33.684684 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:33.685163 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.685796 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.686537 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.686823 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.686923 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.689347 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:33779: BlockingRecv error: recv got EOF from 127.8.204.66:33779 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:33 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:33.891914 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:33.886490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:39691)
0806 23:57:33.886834 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:33.886848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:33.886947 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:33.887182 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:33.887184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:33.887380 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:33.887476 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.887481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.888238 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.888240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.888763 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:33.888770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.889213 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.889215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:33.889275 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.889851 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:33.889870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:33.891805 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
W20260806 23:57:33.892098 21919 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:33.886559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:39691 (local address 127.8.204.126:40697)
0806 23:57:33.886751 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:33.886755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:33.886922 (+ 167us) server_negotiation.cc:407] Connection header received
0806 23:57:33.887031 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:33.887033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:33.887079 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:33.887145 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:33.887614 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.888141 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.888864 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:33.889107 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:33.889190 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:33.891926 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:39691: BlockingRecv error: recv got EOF from 127.8.204.67:39691 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:34.514635 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:34.509515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:42421)
0806 23:57:34.509798 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:34.509810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:34.509895 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:34.510140 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:34.510142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:34.510318 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:34.510409 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.510413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.511076 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.511079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.511595 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.511602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.511965 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.511968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.512051 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.512538 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:34.512549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:34.514498 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
W20260806 23:57:34.514842 21925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:34.509571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:42421 (local address 127.8.204.126:40697)
0806 23:57:34.509841 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:34.509844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:34.509856 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:34.509978 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:34.509981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:34.510022 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:34.510098 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:34.510502 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.510996 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.511698 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.511880 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.511946 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.514663 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:42421: BlockingRecv error: recv got EOF from 127.8.204.65:42421 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:34.695138 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:34.690181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:45729)
0806 23:57:34.690451 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:34.690463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:34.690555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:34.690790 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:34.690792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:34.690981 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:34.691077 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.691082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.691758 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.691760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.692325 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.692332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.692728 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.692731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.692791 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.693260 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:34.693272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:34.695009 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
W20260806 23:57:34.695247 21925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:34.690277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45729 (local address 127.8.204.126:40697)
0806 23:57:34.690411 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:34.690414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:34.690527 (+ 113us) server_negotiation.cc:407] Connection header received
0806 23:57:34.690631 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:34.690633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:34.690673 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:34.690755 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:34.691173 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.691667 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.692422 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.692634 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.692707 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.695136 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:45729: BlockingRecv error: recv got EOF from 127.8.204.66:45729 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:34 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:34.897538 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:34.892603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:36361)
0806 23:57:34.892877 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:34.892894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:34.893017 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:34.893234 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:34.893236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:34.893407 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:34.893505 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.893510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.894212 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.894215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.894735 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:34.894743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.895111 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.895113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:34.895170 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.895717 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:34.895729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:34.897408 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
W20260806 23:57:34.897660 21925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:34.892770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:36361 (local address 127.8.204.126:40697)
0806 23:57:34.892884 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:34.892887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:34.892990 (+ 103us) server_negotiation.cc:407] Connection header received
0806 23:57:34.893083 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:34.893085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:34.893129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:34.893200 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:34.893602 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.894115 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.894829 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:34.895021 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:34.895093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:34.897530 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:36361: BlockingRecv error: recv got EOF from 127.8.204.67:36361 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:35.520372 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:35.515658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:43185)
0806 23:57:35.515937 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:35.515949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:35.516032 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:35.516275 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:35.516277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:35.516443 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:35.516536 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.516541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.517173 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.517175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.517648 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.517654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.518008 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.518010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.518062 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.518587 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:35.518619 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:35.520258 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
W20260806 23:57:35.520502 21931 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:35.515776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:43185 (local address 127.8.204.126:40697)
0806 23:57:35.515992 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:35.515996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:35.516008 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:35.516119 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:35.516121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:35.516162 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:35.516239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:35.516630 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.517093 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.517741 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.517924 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.517988 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.520381 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:43185: BlockingRecv error: recv got EOF from 127.8.204.65:43185 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:35.700858 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:35.695936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:41637)
0806 23:57:35.696212 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:35.696243 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:35.696338 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:35.696586 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:35.696588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:35.696755 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:35.696858 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.696862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.697532 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.697534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.698020 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.698026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.698383 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.698385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.698443 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.699011 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:35.699022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:35.700735 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":65,"threads_started":1}
W20260806 23:57:35.701033 21931 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:35.696023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:41637 (local address 127.8.204.126:40697)
0806 23:57:35.696208 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:35.696212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:35.696310 (+ 98us) server_negotiation.cc:407] Connection header received
0806 23:57:35.696423 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:35.696425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:35.696485 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:35.696547 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:35.696945 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.697446 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.698112 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.698295 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.698361 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.700875 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:41637: BlockingRecv error: recv got EOF from 127.8.204.66:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:35 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:35.903961 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:35.898409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:51853)
0806 23:57:35.898666 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:35.898679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:35.898827 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:35.899051 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:35.899054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:35.899218 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:35.899319 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.899324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.900224 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.900228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.900830 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:35.900841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.901289 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.901292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:35.901356 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.901910 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:35.901924 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:35.903837 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
W20260806 23:57:35.904098 21931 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:35.898471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:51853 (local address 127.8.204.126:40697)
0806 23:57:35.898611 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:35.898615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:35.898773 (+ 158us) server_negotiation.cc:407] Connection header received
0806 23:57:35.898905 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:35.898907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:35.898950 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:35.899007 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:35.899433 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.900090 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.900944 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:35.901176 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:35.901262 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:35.903980 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:51853: BlockingRecv error: recv got EOF from 127.8.204.67:51853 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:36.529101 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:36.521334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:46391)
0806 23:57:36.521635 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:36.521649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:36.521738 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:36.521991 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:36.521993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:36.522173 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:36.522269 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.522274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.522992 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.522995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.523558 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.523568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.523968 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.523971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.524029 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.527410 (+ 3381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:36.527420 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:36.528991 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
W20260806 23:57:36.529269 21936 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:36.521413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46391 (local address 127.8.204.126:40697)
0806 23:57:36.521695 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:36.521698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:36.521712 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:36.521828 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:36.521830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:36.521873 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:36.521951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:36.522370 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.522900 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.523663 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.523873 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.523946 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.529111 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:46391: BlockingRecv error: recv got EOF from 127.8.204.65:46391 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:36.706652 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:36.701547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:47573)
0806 23:57:36.701820 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:36.701833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:36.701929 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:36.702146 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:36.702148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:36.702316 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:36.702405 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.702409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.703151 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.703154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.703697 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.703705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.704100 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.704103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.704238 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.704724 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:36.704735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:36.706493 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
W20260806 23:57:36.706728 21936 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:36.701654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:47573 (local address 127.8.204.126:40697)
0806 23:57:36.701795 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:36.701799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:36.701883 (+ 84us) server_negotiation.cc:407] Connection header received
0806 23:57:36.702004 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:36.702006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:36.702045 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:36.702112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:36.702519 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.703040 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.703797 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.704005 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.704080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.706608 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:47573: BlockingRecv error: recv got EOF from 127.8.204.66:47573 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:36 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:36.910035 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:36.904850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:52211)
0806 23:57:36.905120 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:36.905133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:36.905228 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:36.905488 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:36.905491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:36.905719 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:36.905833 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.905839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.906601 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.906604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.907145 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:36.907153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.907551 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.907554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:36.907610 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.908095 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:36.908105 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:36.909856 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
W20260806 23:57:36.910108 21936 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:36.904986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:52211 (local address 127.8.204.126:40697)
0806 23:57:36.905102 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:36.905105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:36.905198 (+ 93us) server_negotiation.cc:407] Connection header received
0806 23:57:36.905300 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:36.905303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:36.905345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:36.905432 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:36.905952 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.906506 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.907241 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:36.907447 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:36.907534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:36.909964 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:52211: BlockingRecv error: recv got EOF from 127.8.204.67:52211 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:37.537424 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:37.529899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:46167)
0806 23:57:37.530265 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:37.530282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:37.530396 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:37.530661 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:37.530665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:37.530966 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:37.531118 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.531127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.532267 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.532274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.533372 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.533388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.534052 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.534058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.534180 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.534876 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:37.534895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:37.537270 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":76,"threads_started":1}
W20260806 23:57:37.537828 21941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:37.529950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46167 (local address 127.8.204.126:40697)
0806 23:57:37.530196 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:37.530199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:37.530354 (+ 155us) server_negotiation.cc:407] Connection header received
0806 23:57:37.530489 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:37.530491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:37.530530 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:37.530601 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:37.531251 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.532117 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.533513 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.533916 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.534021 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.537701 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:46167: BlockingRecv error: recv got EOF from 127.8.204.65:46167 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:37.712682 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:37.707506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:45305)
0806 23:57:37.707796 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:37.707810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:37.707903 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:37.708139 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:37.708142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:37.708321 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:37.708413 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.708418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.709104 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.709107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.709639 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.709647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.710052 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.710055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.710112 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.710551 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:37.710561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:37.712566 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20260806 23:57:37.712779 21941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:37.707653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45305 (local address 127.8.204.126:40697)
0806 23:57:37.707805 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:37.707809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:37.707872 (+ 63us) server_negotiation.cc:407] Connection header received
0806 23:57:37.707986 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:37.707988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:37.708032 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:37.708099 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:37.708509 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.709000 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.709740 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.709943 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.710011 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.712691 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:45305: BlockingRecv error: recv got EOF from 127.8.204.66:45305 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:37 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:37.916837 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:37.910951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:48329)
0806 23:57:37.911457 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:37.911473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:37.911608 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:37.911941 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:37.911945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:37.912180 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:37.912280 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.912286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.913044 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.913049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.913614 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:37.913622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.914173 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.914176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:37.914279 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.914866 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:37.914878 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:37.916668 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":83,"threads_started":1}
W20260806 23:57:37.916903 21941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:37.911025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:48329 (local address 127.8.204.126:40697)
0806 23:57:37.911178 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:37.911183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:37.911550 (+ 367us) server_negotiation.cc:407] Connection header received
0806 23:57:37.911735 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:37.911741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:37.911806 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:37.911908 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:37.912401 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.912929 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.913740 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:37.914049 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:37.914157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:37.916797 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:48329: BlockingRecv error: recv got EOF from 127.8.204.67:48329 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:38.544023 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:38.538282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:49511)
0806 23:57:38.538559 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:38.538571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:38.538659 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:38.538977 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:38.538979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:38.539218 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:38.539313 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.539319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.540215 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.540218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.540989 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.540998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.541397 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.541399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.541452 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.542033 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:38.542048 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:38.543910 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
W20260806 23:57:38.544298 21945 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:38.538503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:49511 (local address 127.8.204.126:40697)
0806 23:57:38.538763 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:38.538773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:38.538786 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:57:38.538831 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:38.538833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:38.538872 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:38.538941 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:38.539419 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.540045 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.541100 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.541301 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.541376 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.544037 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:49511: BlockingRecv error: recv got EOF from 127.8.204.65:49511 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:38.719026 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:38.713516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:35635)
0806 23:57:38.713819 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:38.713832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:38.713970 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:38.714245 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:38.714248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:38.714465 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:38.714564 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.714569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.715322 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.715327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.715828 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.715835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.716288 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.716291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.716371 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.716990 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:38.717002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:38.718917 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260806 23:57:38.719152 21945 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:38.713565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:35635 (local address 127.8.204.126:40697)
0806 23:57:38.713753 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:38.713758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:38.713907 (+ 149us) server_negotiation.cc:407] Connection header received
0806 23:57:38.714079 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:38.714087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:38.714131 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:38.714225 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:38.714672 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.715204 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.715944 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.716147 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.716273 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.719039 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:35635: BlockingRecv error: recv got EOF from 127.8.204.66:35635 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:38 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:38.922606 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:38.917523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:52687)
0806 23:57:38.917808 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:38.917820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:38.917948 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:38.918207 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:38.918210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:38.918421 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:38.918511 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.918515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.919273 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.919277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.919783 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:38.919791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.920217 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.920221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:38.920302 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.920758 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:38.920769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:38.922498 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
W20260806 23:57:38.922778 21945 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:38.917585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:52687 (local address 127.8.204.126:40697)
0806 23:57:38.917730 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:38.917735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:38.917897 (+ 162us) server_negotiation.cc:407] Connection header received
0806 23:57:38.918045 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:38.918049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:38.918097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:38.918179 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:38.918623 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.919163 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.919899 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:38.920104 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:38.920210 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:38.922617 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:52687: BlockingRecv error: recv got EOF from 127.8.204.67:52687 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:39.549772 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:39.544923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:48209)
0806 23:57:39.545254 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:39.545267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:39.545359 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:39.545610 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:39.545612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:39.545798 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:39.545921 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.545926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.546574 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.546576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.547093 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.547100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.547519 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.547521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.547576 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.548040 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:39.548050 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:39.549666 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
W20260806 23:57:39.549949 21949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:39.545054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:48209 (local address 127.8.204.126:40697)
0806 23:57:39.545287 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:39.545290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:39.545318 (+ 28us) server_negotiation.cc:407] Connection header received
0806 23:57:39.545441 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:39.545443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:39.545485 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:39.545564 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:39.546016 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.546489 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.547193 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.547404 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.547492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.549790 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:48209: BlockingRecv error: recv got EOF from 127.8.204.65:48209 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:39.725225 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:39.719897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:48295)
0806 23:57:39.720233 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:39.720246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:39.720348 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:39.720584 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:39.720587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:39.720763 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:39.720854 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.720859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.721518 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.721521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.722023 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.722030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.722413 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.722416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.722483 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.722984 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:39.722995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:39.725119 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
W20260806 23:57:39.725382 21949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:39.719961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48295 (local address 127.8.204.126:40697)
0806 23:57:39.720098 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:39.720102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:39.720317 (+ 215us) server_negotiation.cc:407] Connection header received
0806 23:57:39.720429 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:39.720431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:39.720475 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:39.720546 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:39.720950 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.721435 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.722125 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.722324 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.722389 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.725237 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:48295: BlockingRecv error: recv got EOF from 127.8.204.66:48295 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:39 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:39.928293 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:39.923450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:45407)
0806 23:57:39.923775 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:39.923787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:39.923884 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:39.924124 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:39.924126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:39.924312 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:39.924405 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.924410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.925083 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.925085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.925566 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:39.925572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.925968 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.925971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:39.926030 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.926504 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:39.926515 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:39.928183 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
W20260806 23:57:39.928471 21949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:39.923509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:45407 (local address 127.8.204.126:40697)
0806 23:57:39.923659 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:39.923662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:39.923858 (+ 196us) server_negotiation.cc:407] Connection header received
0806 23:57:39.923965 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:39.923967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:39.924014 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:39.924083 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:39.924515 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.924999 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.925655 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:39.925874 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:39.925945 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:39.928355 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:45407: BlockingRecv error: recv got EOF from 127.8.204.67:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:40.556173 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:40.550771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:41051)
0806 23:57:40.551072 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:40.551085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:40.551217 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:40.551520 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:40.551522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:40.551709 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:40.551828 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.551837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.552577 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.552582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.553141 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.553150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.553577 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.553579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.553636 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.554136 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:40.554146 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:40.556061 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":118,"threads_started":1}
W20260806 23:57:40.556361 21957 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:40.550881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41051 (local address 127.8.204.126:40697)
0806 23:57:40.551116 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:40.551119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:40.551134 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:40.551311 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:40.551314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:40.551358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:40.551438 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:40.551939 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.552463 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.553256 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.553479 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.553557 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.556221 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:41051: BlockingRecv error: recv got EOF from 127.8.204.65:41051 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:40.731499 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:40.726015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:52417)
0806 23:57:40.726314 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:40.726325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:40.726457 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:40.726724 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:40.726727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:40.726926 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:40.727014 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.727019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.727836 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.727841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.728338 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.728345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.728761 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.728764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.728848 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.729370 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:40.729394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:40.731395 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
W20260806 23:57:40.731650 21957 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:40.726072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:52417 (local address 127.8.204.126:40697)
0806 23:57:40.726254 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:40.726258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:40.726400 (+ 142us) server_negotiation.cc:407] Connection header received
0806 23:57:40.726549 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:40.726552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:40.726601 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:40.726681 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:40.727123 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.727735 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.728449 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.728651 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.728746 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.731511 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:52417: BlockingRecv error: recv got EOF from 127.8.204.66:52417 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:40 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:40.934420 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:40.929278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:55245)
0806 23:57:40.929558 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:40.929571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:40.929665 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:40.929910 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:40.929933 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:40.930135 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:40.930232 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.930238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.930904 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.930906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.931480 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:40.931490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.931888 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.931891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:40.931951 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.932502 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:40.932513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:40.934307 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
W20260806 23:57:40.934613 21957 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:40.929445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:55245 (local address 127.8.204.126:40697)
0806 23:57:40.929566 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:40.929569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:40.929637 (+ 68us) server_negotiation.cc:407] Connection header received
0806 23:57:40.929758 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:40.929761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:40.929803 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:40.929870 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:40.930331 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.930816 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.931584 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:40.931791 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:40.931868 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:40.934477 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:55245: BlockingRecv error: recv got EOF from 127.8.204.67:55245 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:41.565649 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:41.557011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:46023)
0806 23:57:41.557338 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:41.557351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:41.557459 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:41.557689 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:41.557691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:41.557864 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:41.557957 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.557962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.558629 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.558631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.559250 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.559258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.559793 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.559797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.559885 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.563615 (+ 3730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:41.563627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:41.565482 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
W20260806 23:57:41.565718 21961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:41.557061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46023 (local address 127.8.204.126:40697)
0806 23:57:41.557328 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:41.557334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:41.557431 (+ 97us) server_negotiation.cc:407] Connection header received
0806 23:57:41.557535 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:41.557537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:41.557578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:41.557652 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:41.558056 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.558529 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.559345 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.559661 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.559783 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.565595 (+ 5812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:46023: BlockingRecv error: recv got EOF from 127.8.204.65:46023 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:41.737144 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:41.732199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:36771)
0806 23:57:41.732451 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:41.732463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:41.732557 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:41.732748 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:41.732750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:41.732920 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:41.733009 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.733013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.733688 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.733691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.734221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.734228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.734628 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.734631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.734691 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.735184 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:41.735195 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:41.737021 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
W20260806 23:57:41.737358 21961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:41.732281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:36771 (local address 127.8.204.126:40697)
0806 23:57:41.732411 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:41.732415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:41.732526 (+ 111us) server_negotiation.cc:407] Connection header received
0806 23:57:41.732629 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:41.732631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:41.732671 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:41.732735 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:41.733096 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.733594 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.734320 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.734524 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.734599 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.737147 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:36771: BlockingRecv error: recv got EOF from 127.8.204.66:36771 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:41 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:41.940522 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:41.935203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:46877)
0806 23:57:41.935450 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:41.935462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:41.935593 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:41.935847 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:41.935850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:41.936051 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:41.936147 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.936153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.936860 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.936864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.937382 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:41.937390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.937826 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.937829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:41.937915 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.938393 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:41.938438 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:41.940384 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20260806 23:57:41.940586 21961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:41.935244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:46877 (local address 127.8.204.126:40697)
0806 23:57:41.935378 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:41.935383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:41.935533 (+ 150us) server_negotiation.cc:407] Connection header received
0806 23:57:41.935701 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:41.935703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:41.935746 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:41.935823 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:41.936260 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.936747 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.937506 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:41.937713 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:41.937811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:41.940498 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:46877: BlockingRecv error: recv got EOF from 127.8.204.67:46877 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:42.571564 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:42.566670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:33409)
0806 23:57:42.566953 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:42.566967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:42.567049 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:42.567272 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:42.567275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:42.567452 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:42.567548 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.567553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.568223 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.568226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.568794 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.568801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.569197 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.569199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.569267 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.569733 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:42.569743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:42.571423 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
W20260806 23:57:42.571760 21966 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:42.566771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:33409 (local address 127.8.204.126:40697)
0806 23:57:42.567008 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:42.567012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:42.567026 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:42.567117 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:42.567119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:42.567157 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:42.567233 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:42.567650 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.568137 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.568895 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.569103 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.569176 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.571591 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:33409: BlockingRecv error: recv got EOF from 127.8.204.65:33409 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:42.743165 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:42.738045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:47689)
0806 23:57:42.738287 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:42.738300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:42.738391 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:42.738723 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:42.738727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:42.738919 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:42.739061 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.739067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.739821 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.739825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.740384 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.740393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.740819 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.740822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.740902 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.741334 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:42.741345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:42.743051 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
W20260806 23:57:42.743312 21966 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:42.738118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:47689 (local address 127.8.204.126:40697)
0806 23:57:42.738248 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:42.738251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:42.738348 (+ 97us) server_negotiation.cc:407] Connection header received
0806 23:57:42.738489 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:42.738494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:42.738589 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:42.738666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:42.739164 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.739723 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.740503 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.740706 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.740801 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.743166 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:47689: BlockingRecv error: recv got EOF from 127.8.204.66:47689 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:42 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:42.946223 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:42.941307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:41287)
0806 23:57:42.941560 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:42.941572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:42.941651 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:42.941873 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:42.941875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:42.942053 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:42.942145 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.942150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.942796 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.942799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.943308 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:42.943316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.943712 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.943714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:42.943776 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.944255 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:42.944266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:42.946067 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
W20260806 23:57:42.946280 21966 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:42.941445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:41287 (local address 127.8.204.126:40697)
0806 23:57:42.941592 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:42.941597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:42.941612 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:57:42.941730 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:42.941733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:42.941775 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:42.941838 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:42.942239 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.942718 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.943404 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:42.943600 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:42.943689 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:42.946175 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:41287: BlockingRecv error: recv got EOF from 127.8.204.67:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:43.578020 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:43.572331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:43499)
0806 23:57:43.572734 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:43.572752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:43.572863 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:43.573213 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:43.573217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:43.573459 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:43.573632 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.573644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.574436 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.574439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.575127 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.575137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.575533 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.575535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.575591 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.576095 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:43.576110 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:43.577907 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":98,"threads_started":1}
W20260806 23:57:43.578210 21970 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:43.572448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:43499 (local address 127.8.204.126:40697)
0806 23:57:43.572735 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:43.572739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:43.572839 (+ 100us) server_negotiation.cc:407] Connection header received
0806 23:57:43.572966 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:43.572969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:43.573020 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:43.573148 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:43.573785 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.574342 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.575246 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.575442 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.575516 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.578036 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:43499: BlockingRecv error: recv got EOF from 127.8.204.65:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:43.749267 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:43.743984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:41039)
0806 23:57:43.744293 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:43.744309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:43.744438 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:43.744761 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:43.744766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:43.745013 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:43.745139 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.745144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.745917 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.745922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.746427 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.746435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.746925 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.746929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.747010 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.747485 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:43.747496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:43.749115 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
W20260806 23:57:43.749300 21970 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:43.744044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:41039 (local address 127.8.204.126:40697)
0806 23:57:43.744204 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:43.744208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:43.744399 (+ 191us) server_negotiation.cc:407] Connection header received
0806 23:57:43.744537 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:43.744541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:43.744612 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:43.744715 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:43.745248 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.745811 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.746542 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.746819 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.746920 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.749215 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:41039: BlockingRecv error: recv got EOF from 127.8.204.66:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:43 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:43.952241 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:43.947083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:37667)
0806 23:57:43.947367 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:43.947380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:43.947469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:43.947693 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:43.947695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:43.947863 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:43.947955 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.947960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.948630 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.948632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.949224 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:43.949235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.949728 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.949732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:43.949807 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.950339 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:43.950350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:43.952115 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260806 23:57:43.952294 21970 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:43.947272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:37667 (local address 127.8.204.126:40697)
0806 23:57:43.947413 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:43.947416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:43.947428 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:43.947549 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:43.947551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:43.947591 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:43.947655 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:43.948059 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.948530 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.949336 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:43.949615 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:43.949698 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:43.952208 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:37667: BlockingRecv error: recv got EOF from 127.8.204.67:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:44.583843 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:44.578744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:33459)
0806 23:57:44.579010 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:44.579025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:44.579126 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:44.579374 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:44.579377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:44.579541 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:44.579635 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.579639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.580337 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.580340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.580838 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.580845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.581294 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.581297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.581379 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.581907 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:44.581919 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:44.583652 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":64,"threads_started":1}
W20260806 23:57:44.583905 21977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:44.578898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:33459 (local address 127.8.204.126:40697)
0806 23:57:44.579117 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:44.579120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:44.579132 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:44.579196 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:44.579198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:44.579237 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:44.579325 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:44.579738 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.580247 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.580958 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.581165 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.581278 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.583788 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:33459: BlockingRecv error: recv got EOF from 127.8.204.65:33459 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:44.755731 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:44.750041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:49233)
0806 23:57:44.750345 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:44.750360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:44.750498 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:44.750764 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:44.750767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:44.750983 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:44.751077 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.751083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.751780 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.751785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.752476 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.752487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.753080 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.753082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.753182 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.753749 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:44.753761 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:44.755624 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
W20260806 23:57:44.755890 21977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:44.750215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49233 (local address 127.8.204.126:40697)
0806 23:57:44.750348 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:44.750353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:44.750438 (+ 85us) server_negotiation.cc:407] Connection header received
0806 23:57:44.750587 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:44.750590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:44.750638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:44.750736 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:44.751180 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.751648 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.752609 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.752939 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.753075 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.755743 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:49233: BlockingRecv error: recv got EOF from 127.8.204.66:49233 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:44 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:44.958781 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:44.953019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:53393)
0806 23:57:44.953309 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:44.953321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:44.953461 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:44.953742 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:44.953746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:44.953959 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:44.954052 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.954058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.954872 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.954877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.955421 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:44.955429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.955913 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.955917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:44.956012 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.956580 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:44.956599 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:44.958592 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20260806 23:57:44.958899 21977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:44.953080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:53393 (local address 127.8.204.126:40697)
0806 23:57:44.953247 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:44.953253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:44.953401 (+ 148us) server_negotiation.cc:407] Connection header received
0806 23:57:44.953564 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:44.953567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:44.953617 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:44.953714 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:44.954177 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.954750 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.955550 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:44.955789 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:44.955896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:44.958788 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:53393: BlockingRecv error: recv got EOF from 127.8.204.67:53393 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:45.589789 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:45.584746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:36387)
0806 23:57:45.585050 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:45.585064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:45.585163 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:45.585399 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:45.585401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:45.585571 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:45.585664 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.585669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.586352 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.586354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.586989 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.587002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.587438 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.587441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.587498 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.587955 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:45.587966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:45.589678 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
W20260806 23:57:45.589922 21982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:45.584787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:36387 (local address 127.8.204.126:40697)
0806 23:57:45.585103 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:45.585106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:45.585118 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:57:45.585247 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:45.585249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:45.585291 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:45.585360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:45.585760 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.586262 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.587104 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.587341 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.587421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.589796 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:36387: BlockingRecv error: recv got EOF from 127.8.204.65:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":63,"threads_started":1}
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:45.761432 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:45.756467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:49449)
0806 23:57:45.756746 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:45.756759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:45.756849 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:45.757074 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:45.757076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:45.757270 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:45.757362 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.757367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.758039 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.758042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.758565 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.758572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.758981 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.758984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.759042 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.759543 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:45.759554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:45.761312 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
W20260806 23:57:45.761590 21982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:45.756629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49449 (local address 127.8.204.126:40697)
0806 23:57:45.756754 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:45.756763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:45.756806 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:57:45.756929 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:45.756931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:45.756974 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:45.757037 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:45.757458 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.757952 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.758663 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.758894 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.758963 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.761437 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:49449: BlockingRecv error: recv got EOF from 127.8.204.66:49449 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:45 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:45.965133 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:45.959756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:60785)
0806 23:57:45.960040 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:45.960052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:45.960155 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:45.960379 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:45.960381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:45.960548 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:45.960655 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.960659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.961361 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.961364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.961951 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:45.961961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.962542 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.962546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:45.962634 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.963113 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:45.963125 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:45.964929 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
W20260806 23:57:45.965170 21982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:45.959817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:60785 (local address 127.8.204.126:40697)
0806 23:57:45.959983 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:45.959987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:45.960130 (+ 143us) server_negotiation.cc:407] Connection header received
0806 23:57:45.960232 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:45.960234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:45.960278 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:45.960336 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:45.960760 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.961241 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.962095 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:45.962424 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:45.962535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:45.965074 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:60785: BlockingRecv error: recv got EOF from 127.8.204.67:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:46.601034 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:46.590675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:38429)
0806 23:57:46.590998 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:46.591011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:46.591107 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:46.591345 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:46.591347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:46.591528 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:46.591635 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.591640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.592376 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.592378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.592923 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.592930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.593349 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.593351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.593408 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.598625 (+ 5217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:46.598640 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:46.600859 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
W20260806 23:57:46.601110 21988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:46.590804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:38429 (local address 127.8.204.126:40697)
0806 23:57:46.591103 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:46.591106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:46.591120 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:57:46.591177 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:46.591179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:46.591227 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:46.591305 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:46.591739 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.592282 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.593023 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.593252 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.593324 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.600985 (+ 7661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:38429: BlockingRecv error: recv got EOF from 127.8.204.65:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:46.767427 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:46.762212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:38781)
0806 23:57:46.762463 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:46.762475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:46.762571 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:46.762818 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:46.762820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:46.763022 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:46.763133 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.763139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.763881 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.763883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.764449 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.764457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.764860 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.764863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.764923 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.765409 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:46.765420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:46.767303 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
W20260806 23:57:46.767518 21988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:46.762273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:38781 (local address 127.8.204.126:40697)
0806 23:57:46.762399 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:46.762403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:46.762541 (+ 138us) server_negotiation.cc:407] Connection header received
0806 23:57:46.762636 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:46.762638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:46.762679 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:46.762764 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:46.763233 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.763787 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.764550 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.764763 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.764836 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.767430 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:38781: BlockingRecv error: recv got EOF from 127.8.204.66:38781 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:46 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.67@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:46.971415 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:46.965980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.67:34241)
0806 23:57:46.966299 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:46.966314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:46.966425 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:46.966688 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:46.966691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:46.967070 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:46.967210 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.967217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.967959 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.967962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.968471 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:46.968478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.968849 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.968851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:46.968909 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.969387 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:46.969397 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:46.971295 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
W20260806 23:57:46.971560 21988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:46.966106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:34241 (local address 127.8.204.126:40697)
0806 23:57:46.966243 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:46.966249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:46.966401 (+ 152us) server_negotiation.cc:407] Connection header received
0806 23:57:46.966508 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:46.966511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:46.966567 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:46.966658 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:46.967341 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.967866 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.968565 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:46.968758 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:46.968827 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:46.971413 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.67:34241: BlockingRecv error: recv got EOF from 127.8.204.67:34241 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 23:57:47 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:47 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.65@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:47.607839 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:47.601929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.65:45237)
0806 23:57:47.602204 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:47.602217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:47.602306 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:47.602525 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:47.602527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:47.602700 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:47.602847 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:47.602853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.603525 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.603528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:47.604080 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:47.604088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.604571 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.604575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:47.604663 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:47.605398 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:47.605430 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:47.607710 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
W20260806 23:57:47.608011 21992 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:47.601970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:45237 (local address 127.8.204.126:40697)
0806 23:57:47.602208 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:47.602212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:47.602279 (+ 67us) server_negotiation.cc:407] Connection header received
0806 23:57:47.602378 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:47.602380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:47.602421 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:47.602491 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:47.602953 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.603438 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.604175 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.604452 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.604563 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:47.607849 (+ 3286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.65:45237: BlockingRecv error: recv got EOF from 127.8.204.65:45237 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
Aug 06 23:57:47 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:57:47 dist-test-slave-39l8 krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.66@KRBTEST.COM for oryx/127.8.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:57:47.773833 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:47.768350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:40697 (local address 127.8.204.66:42289)
0806 23:57:47.768662 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:47.768677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:47.768809 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:47.769094 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:47.769097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:47.769313 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:47.769405 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:47.769410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.770152 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.770157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:47.770695 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:47.770761 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.771223 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.771226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:47.771311 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:47.771883 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:47.771895 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:47.773663 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.126:40697: Server oryx/127.8.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
W20260806 23:57:47.773893 21992 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:47.768465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:42289 (local address 127.8.204.126:40697)
0806 23:57:47.768638 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:47.768643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:47.768762 (+ 119us) server_negotiation.cc:407] Connection header received
0806 23:57:47.768915 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:47.768919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:47.768971 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:47.769054 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:47.769528 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.770044 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.770878 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:47.771098 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:47.771216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:47.773787 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.66:42289: BlockingRecv error: recv got EOF from 127.8.204.66:42289 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:57:47.880565 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21359
I20260806 23:57:47.897109 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21559
I20260806 23:57:47.905740 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21745
I20260806 23:57:47.916827 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21267
2026-08-06T23:57:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18088 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T23:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:47Z Disabled control of system clock
I20260806 23:57:47.951781 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44303
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 23:57:48.060596 21999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:48.060815 21999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:48.060847 21999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:48.064543 21999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:48.064584 21999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:48.064600 21999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:48.064616 21999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:48.064631 21999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:48.068940 21999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:48.069828 21999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:48.071035 21999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:48.079210 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.079332 21999 server_base.cc:1034] running on GCE node
W20260806 23:57:48.079193 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.079193 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.079994 21999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:48.081035 21999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:48.082227 21999 hybrid_clock.cc:648] HybridClock initialized: now 1786060668082212 us; error 32 us; skew 500 ppm
I20260806 23:57:48.083957 21999 webserver.cc:460] Webserver started at http://127.8.204.126:41019/ using document root <none> and password file <none>
I20260806 23:57:48.084529 21999 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:48.084632 21999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:48.084867 21999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:48.086921 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "d13118ab4d254cd1b3587eefba0781ec"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "2e469e2b5f50d3064a814ee6298955f2"
server_key_iv: "c12826e0ff734d2a6b0a0906e9cd2bce"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.087481 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "d13118ab4d254cd1b3587eefba0781ec"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "2e469e2b5f50d3064a814ee6298955f2"
server_key_iv: "c12826e0ff734d2a6b0a0906e9cd2bce"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.090967 21999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:57:48.093132 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.094209 21999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:48.094352 21999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "d13118ab4d254cd1b3587eefba0781ec"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "2e469e2b5f50d3064a814ee6298955f2"
server_key_iv: "c12826e0ff734d2a6b0a0906e9cd2bce"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.094460 21999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:48.110236 21999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:48.110893 21999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:48.111097 21999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:48.117923 21999 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44303
I20260806 23:57:48.117930 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44303 every 8 connection(s)
I20260806 23:57:48.118816 21999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:48.121467 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:48.126653 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec: Bootstrap starting.
I20260806 23:57:48.127022 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 21999
I20260806 23:57:48.127143 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:48.127439 9009 external_mini_cluster.cc:1428] Setting key 046cb401757af92c60ab64cc03a37fd8
I20260806 23:57:48.129300 22084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:48.130137 22084 log.cc:824] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:48.131947 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec: No bootstrap required, opened a new log
I20260806 23:57:48.135455 22084 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } }
I20260806 23:57:48.135684 22084 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:48.135797 22084 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13118ab4d254cd1b3587eefba0781ec, State: Initialized, Role: FOLLOWER
I20260806 23:57:48.136257 22084 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } }
I20260806 23:57:48.136426 22084 raft_consensus.cc:388] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:48.136510 22084 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:48.136633 22084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:48.136894 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.128783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.8.204.126:44303)
0806 23:57:48.129434 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.129451 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.129491 (+ 40us) server_negotiation.cc:407] Connection header received
0806 23:57:48.130184 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:48.130202 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:48.130681 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:48.131088 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:48.131487 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.132511 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.133409 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.133884 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.134057 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.134994 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:48.134999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:48.135010 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:48.135379 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:48.135454 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:48.135473 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:48.135582 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:48.135750 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 23:57:48.136007 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260806 23:57:48.137528 22084 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } }
I20260806 23:57:48.137907 22084 leader_election.cc:302] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d13118ab4d254cd1b3587eefba0781ec; no voters:
I20260806 23:57:48.138237 22084 leader_election.cc:288] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:48.138340 22090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:48.138648 22090 raft_consensus.cc:686] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [term 1 LEADER]: Becoming Leader. State: Replica: d13118ab4d254cd1b3587eefba0781ec, State: Running, Role: LEADER
I20260806 23:57:48.139016 22090 consensus_queue.cc:237] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } }
I20260806 23:57:48.139427 22084 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:48.140183 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d13118ab4d254cd1b3587eefba0781ec. Latest consensus state: current_term: 1 leader_uuid: "d13118ab4d254cd1b3587eefba0781ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } } }
I20260806 23:57:48.140355 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:48.140648 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d13118ab4d254cd1b3587eefba0781ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13118ab4d254cd1b3587eefba0781ec" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44303 } } }
I20260806 23:57:48.140763 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:48.140800 22099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:48.143851 22099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:48.154059 22099 catalog_manager.cc:1347] Generated new cluster ID: a5d7aaddf7ec4a299b0241f1267671e2
I20260806 23:57:48.154141 22099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:48.172830 22099 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:48.173668 22099 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:48.180718 22099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d13118ab4d254cd1b3587eefba0781ec: Generated new TSK 0
I20260806 23:57:48.181371 22099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:48.195013 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--builtin_ntp_servers=127.8.204.84:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 23:57:48.317365 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:48.317584 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:48.317615 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:48.321359 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:48.321403 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:48.321450 22112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:48.325790 22112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:48.326730 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:48.327924 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:48.337152 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.337214 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.337270 22112 server_base.cc:1034] running on GCE node
W20260806 23:57:48.337400 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.337744 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:48.338371 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:48.339589 22112 hybrid_clock.cc:648] HybridClock initialized: now 1786060668339548 us; error 47 us; skew 500 ppm
I20260806 23:57:48.341334 22112 webserver.cc:460] Webserver started at http://127.8.204.65:35119/ using document root <none> and password file <none>
I20260806 23:57:48.341989 22112 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:48.342078 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:48.342278 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:48.344307 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "67b8d873aa2845178dafc0387a7f5890"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7252beee9e631e17ad2e51e7df1fe12"
server_key_iv: "cccf465bb6edaf95ebbb929068b5a786"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.344856 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "67b8d873aa2845178dafc0387a7f5890"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7252beee9e631e17ad2e51e7df1fe12"
server_key_iv: "cccf465bb6edaf95ebbb929068b5a786"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.348325 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:48.350577 22134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.351713 22112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:48.351841 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "67b8d873aa2845178dafc0387a7f5890"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7252beee9e631e17ad2e51e7df1fe12"
server_key_iv: "cccf465bb6edaf95ebbb929068b5a786"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.351948 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:48.372704 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:48.373281 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:48.373463 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:48.374010 22112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:48.374915 22112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:48.374989 22112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.375061 22112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:48.375103 22112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.383776 22112 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:44531
I20260806 23:57:48.383819 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:44531 every 8 connection(s)
I20260806 23:57:48.384582 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:48.392619 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.386198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:45815 (local address 127.8.204.126:44303)
0806 23:57:48.386330 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.386334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.387203 (+ 869us) server_negotiation.cc:407] Connection header received
0806 23:57:48.388460 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:48.388463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:48.388508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:48.388579 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:48.389397 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.389901 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.390994 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.391201 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.391278 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.392253 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:48.392255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:48.392257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:48.392268 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:48.392281 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:48.392287 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:48.392339 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:48.392469 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 23:57:48.392513 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:57:48.392908 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22112
I20260806 23:57:48.393025 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:48.393312 9009 external_mini_cluster.cc:1428] Setting key dd0f01c4c3cc1bcb50f8cf3457dbd438
I20260806 23:57:48.393496 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.386460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44303 (local address 127.8.204.65:45815)
0806 23:57:48.387077 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:48.387121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:48.388283 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:48.388731 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:48.388736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:48.388807 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:48.389281 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.389292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.390005 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.390008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.390891 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.390902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.391327 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.391330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.391457 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.392031 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:48.392033 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:48.392068 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:48.392130 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:48.392357 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:48.392364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:48.392368 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:57:48.392454 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:57:48.392639 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
I20260806 23:57:48.394757 22281 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44303
I20260806 23:57:48.394981 22281 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:48.395396 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--builtin_ntp_servers=127.8.204.84:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 23:57:48.395560 22281 heartbeater.cc:511] Master 127.8.204.126:44303 requested a full tablet report, sending...
I20260806 23:57:48.396911 22037 ts_manager.cc:194] Registered new tserver with Master: 67b8d873aa2845178dafc0387a7f5890 (127.8.204.65:44531)
I20260806 23:57:48.398109 22037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.65:45815
W20260806 23:57:48.501839 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:48.502036 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:48.502066 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:48.505685 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:48.505722 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:48.505787 22285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:48.509738 22285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:48.510609 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:48.511896 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:57:48.520756 22285 server_base.cc:1034] running on GCE node
W20260806 23:57:48.520763 22299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.520802 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.520888 22295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.521282 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:48.521857 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:48.523032 22285 hybrid_clock.cc:648] HybridClock initialized: now 1786060668523021 us; error 32 us; skew 500 ppm
I20260806 23:57:48.524755 22285 webserver.cc:460] Webserver started at http://127.8.204.66:46309/ using document root <none> and password file <none>
I20260806 23:57:48.525303 22285 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:48.525413 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:48.525636 22285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:48.527499 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "b9ffbabb68d74ba497201bc0b3bfe3cf"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "7e150b8f0525c5997b3edf2e8d9159bc"
server_key_iv: "276fe206dd7115e754ebcb0a82352c03"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.528013 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "b9ffbabb68d74ba497201bc0b3bfe3cf"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "7e150b8f0525c5997b3edf2e8d9159bc"
server_key_iv: "276fe206dd7115e754ebcb0a82352c03"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.531440 22285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 23:57:48.533872 22307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.535065 22285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 23:57:48.535205 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "b9ffbabb68d74ba497201bc0b3bfe3cf"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "7e150b8f0525c5997b3edf2e8d9159bc"
server_key_iv: "276fe206dd7115e754ebcb0a82352c03"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.535323 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:48.546953 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:48.547508 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:48.547685 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:48.548195 22285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:48.549036 22285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:48.549115 22285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.549194 22285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:48.549242 22285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.557798 22285 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:44065
I20260806 23:57:48.557822 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:44065 every 8 connection(s)
I20260806 23:57:48.558627 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:48.562275 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22285
I20260806 23:57:48.562403 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:48.562690 9009 external_mini_cluster.cc:1428] Setting key 543f21a52f0fefb35114f504a7bb7396
I20260806 23:57:48.564760 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--builtin_ntp_servers=127.8.204.84:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 23:57:48.567658 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.560345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45467 (local address 127.8.204.126:44303)
0806 23:57:48.560481 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.560484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.561440 (+ 956us) server_negotiation.cc:407] Connection header received
0806 23:57:48.562919 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:48.562923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:48.562982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:48.563086 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:48.564021 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.564577 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.565771 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.565994 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.566107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.567180 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:48.567184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:48.567186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:48.567199 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:48.567225 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:48.567232 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:48.567318 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:48.567463 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 23:57:48.567520 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:57:48.568583 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.560677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44303 (local address 127.8.204.66:45467)
0806 23:57:48.561305 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:48.561339 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:48.562660 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:48.563223 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:48.563229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:48.563309 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:48.563871 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.563889 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.564723 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.564727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.565640 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.565650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.566165 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.566170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.566316 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.566925 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:48.566927 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:48.566962 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:48.567018 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:48.567320 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:48.567332 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:48.567339 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:57:48.567462 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 23:57:48.567697 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260806 23:57:48.569684 22454 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44303
I20260806 23:57:48.569906 22454 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:48.570390 22454 heartbeater.cc:511] Master 127.8.204.126:44303 requested a full tablet report, sending...
I20260806 23:57:48.571461 22037 ts_manager.cc:194] Registered new tserver with Master: b9ffbabb68d74ba497201bc0b3bfe3cf (127.8.204.66:44065)
I20260806 23:57:48.572019 22037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.66:45467
W20260806 23:57:48.672003 22458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:48.672246 22458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:48.672281 22458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:48.676383 22458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:48.676425 22458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:48.676471 22458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:48.680711 22458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:48.681643 22458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:48.682960 22458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:48.692229 22468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.692215 22465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.692214 22464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.692577 22458 server_base.cc:1034] running on GCE node
I20260806 23:57:48.692945 22458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:48.693584 22458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:48.694787 22458 hybrid_clock.cc:648] HybridClock initialized: now 1786060668694777 us; error 27 us; skew 500 ppm
I20260806 23:57:48.696668 22458 webserver.cc:460] Webserver started at http://127.8.204.67:41289/ using document root <none> and password file <none>
I20260806 23:57:48.697232 22458 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:48.697398 22458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:48.697640 22458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:48.699879 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "475d87c203d449ebb9e95aacd549e8ed"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7263038d1c14553fd3f756dac978995"
server_key_iv: "50a0c3652e6aaea7ad32cdaa56966bdf"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.700425 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "475d87c203d449ebb9e95aacd549e8ed"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7263038d1c14553fd3f756dac978995"
server_key_iv: "50a0c3652e6aaea7ad32cdaa56966bdf"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.704016 22458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:48.706460 22478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.707608 22458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:48.707736 22458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "475d87c203d449ebb9e95aacd549e8ed"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "f7263038d1c14553fd3f756dac978995"
server_key_iv: "50a0c3652e6aaea7ad32cdaa56966bdf"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.707839 22458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:48.722635 22458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:48.723333 22458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:48.723524 22458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:48.724069 22458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:48.724999 22458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:48.725077 22458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.725143 22458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:48.725198 22458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.734131 22458 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:35349
I20260806 23:57:48.734180 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:35349 every 8 connection(s)
I20260806 23:57:48.735092 22458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:48.742647 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22458
I20260806 23:57:48.742794 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:48.743049 9009 external_mini_cluster.cc:1428] Setting key dd0c1a12fbeb6f79d7155f4786bda3bf
I20260806 23:57:48.743935 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.736813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:48307 (local address 127.8.204.126:44303)
0806 23:57:48.736946 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.736950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.737910 (+ 960us) server_negotiation.cc:407] Connection header received
0806 23:57:48.739245 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:48.739247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:48.739293 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:48.739364 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:48.740217 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.740783 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.741912 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.742108 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.742184 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.743333 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:48.743343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:48.743345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:48.743355 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:48.743372 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:48.743379 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:48.743435 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:48.743753 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 23:57:48.743802 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:57:48.744693 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.737181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44303 (local address 127.8.204.67:48307)
0806 23:57:48.737756 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:48.737791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:48.739054 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:48.739541 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:48.739547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:48.739618 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:48.740095 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.740106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.740908 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.740910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.741814 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:48.741823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:48.742228 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:48.742230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:48.742353 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:48.743066 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:48.743068 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:48.743102 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:48.743197 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:48.743467 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:48.743477 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:48.743483 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:57:48.743606 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 23:57:48.743818 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260806 23:57:48.745806 22621 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44303
I20260806 23:57:48.745991 22621 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:48.746409 22621 heartbeater.cc:511] Master 127.8.204.126:44303 requested a full tablet report, sending...
I20260806 23:57:48.747459 22037 ts_manager.cc:194] Registered new tserver with Master: 475d87c203d449ebb9e95aacd549e8ed (127.8.204.67:35349)
I20260806 23:57:48.747946 22037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.67:48307
I20260806 23:57:48.756732 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 23:57:48.759305 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:48.758518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.8.204.126:44303)
0806 23:57:48.758621 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.758624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.758813 (+ 189us) server_negotiation.cc:407] Connection header received
0806 23:57:48.758884 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:48.758886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:48.758920 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:48.758981 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:48.759171 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49700: BlockingRecv error: recv got EOF from 127.0.0.1:49700 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:57:48.759717 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22112
I20260806 23:57:48.774243 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22285
I20260806 23:57:48.786378 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22458
I20260806 23:57:48.799616 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 21999
2026-08-06T23:57:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T23:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:48Z Disabled control of system clock
I20260806 23:57:48.831794 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44705
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 23:57:48.942018 22633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:48.942240 22633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:48.942276 22633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:48.946108 22633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:48.946156 22633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:48.946169 22633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:48.946184 22633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:48.946197 22633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:48.950557 22633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:48.951475 22633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:48.952735 22633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:48.960803 22640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:48.960811 22641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.960953 22633 server_base.cc:1034] running on GCE node
W20260806 23:57:48.960986 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:48.961617 22633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:48.962500 22633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:48.963651 22633 hybrid_clock.cc:648] HybridClock initialized: now 1786060668963623 us; error 42 us; skew 500 ppm
I20260806 23:57:48.965363 22633 webserver.cc:460] Webserver started at http://127.8.204.126:39819/ using document root <none> and password file <none>
I20260806 23:57:48.965874 22633 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:48.965930 22633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:48.966156 22633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:48.968088 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "938fdbf67de049b48024702d19343417"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "34b28c306742f61f1051ac2965d70e00"
server_key_iv: "d2c673f6eccc6710e9ea53ebe98ac366"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.968652 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "938fdbf67de049b48024702d19343417"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "34b28c306742f61f1051ac2965d70e00"
server_key_iv: "d2c673f6eccc6710e9ea53ebe98ac366"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.971994 22633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:48.974195 22650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:48.975355 22633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:48.975509 22633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "938fdbf67de049b48024702d19343417"
format_stamp: "Formatted at 2026-08-06 23:57:48 on dist-test-slave-39l8"
server_key: "34b28c306742f61f1051ac2965d70e00"
server_key_iv: "d2c673f6eccc6710e9ea53ebe98ac366"
server_key_version: "encryptionkey@0"
I20260806 23:57:48.975615 22633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:48.979837 22633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:48.980381 22633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:48.980551 22633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:48.987219 22633 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44705
I20260806 23:57:48.987246 22715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44705 every 8 connection(s)
I20260806 23:57:48.988040 22633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:48.991046 22716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:48.996016 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417: Bootstrap starting.
I20260806 23:57:48.997328 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22633
I20260806 23:57:48.997460 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:48.997728 9009 external_mini_cluster.cc:1428] Setting key 1e98a61a4d68dc353a7b86034ffd242a
I20260806 23:57:48.998409 22716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:48.999333 22716 log.cc:824] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:49.001180 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417: No bootstrap required, opened a new log
I20260806 23:57:49.002826 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:48.998854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.8.204.126:44705)
0806 23:57:48.999446 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:48.999484 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:48.999529 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:57:49.000292 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:49.000310 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:49.000567 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:49.000886 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:49.001626 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:49.001631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:49.001642 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:49.001653 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:49.001709 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:49.001725 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:49.001841 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:49.001936 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 23:57:49.002154 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260806 23:57:49.004789 22716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } }
I20260806 23:57:49.005009 22716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:49.005122 22716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938fdbf67de049b48024702d19343417, State: Initialized, Role: FOLLOWER
I20260806 23:57:49.005541 22716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } }
I20260806 23:57:49.005682 22716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:49.005744 22716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:49.005857 22716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:49.006669 22716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } }
I20260806 23:57:49.007046 22716 leader_election.cc:302] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938fdbf67de049b48024702d19343417; no voters:
I20260806 23:57:49.007349 22716 leader_election.cc:288] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:49.007448 22722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:49.007696 22722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [term 1 LEADER]: Becoming Leader. State: Replica: 938fdbf67de049b48024702d19343417, State: Running, Role: LEADER
I20260806 23:57:49.008038 22722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } }
I20260806 23:57:49.008468 22716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:49.009874 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938fdbf67de049b48024702d19343417. Latest consensus state: current_term: 1 leader_uuid: "938fdbf67de049b48024702d19343417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } } }
I20260806 23:57:49.009889 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938fdbf67de049b48024702d19343417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938fdbf67de049b48024702d19343417" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44705 } } }
I20260806 23:57:49.010028 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:49.010030 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:49.010454 22741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:49.013293 22741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:49.018450 22741 catalog_manager.cc:1347] Generated new cluster ID: 3aa953fc54874458b4f55efbd6609aa7
I20260806 23:57:49.018513 22741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:49.026885 22741 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:49.027145 22741 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:49.038770 22741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938fdbf67de049b48024702d19343417: Generated new TSK 0
I20260806 23:57:49.039335 22741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:57:49.047223 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--builtin_ntp_servers=127.8.204.84:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 23:57:49.155351 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:49.155586 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:49.155638 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:49.159484 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:49.159523 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:49.159602 22750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:49.164004 22750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:49.164934 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:49.166134 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:49.174961 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:49.174997 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:49.175027 22750 server_base.cc:1034] running on GCE node
W20260806 23:57:49.174959 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:49.175624 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:49.176119 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:49.177270 22750 hybrid_clock.cc:648] HybridClock initialized: now 1786060669177245 us; error 40 us; skew 500 ppm
I20260806 23:57:49.178934 22750 webserver.cc:460] Webserver started at http://127.8.204.65:46235/ using document root <none> and password file <none>
I20260806 23:57:49.179464 22750 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:49.179519 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:49.179661 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:49.181401 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "91e4e461027e4ca1838f52810e3e2a60"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "cbffc2b49835c49c7c838cc3671d2d9a"
server_key_iv: "cb9eaa6c4818f6be3e2c04103642a96c"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.181852 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "91e4e461027e4ca1838f52810e3e2a60"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "cbffc2b49835c49c7c838cc3671d2d9a"
server_key_iv: "cb9eaa6c4818f6be3e2c04103642a96c"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.185144 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 23:57:49.187312 22767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.188373 22750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:57:49.188495 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "91e4e461027e4ca1838f52810e3e2a60"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "cbffc2b49835c49c7c838cc3671d2d9a"
server_key_iv: "cb9eaa6c4818f6be3e2c04103642a96c"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.188588 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:49.192874 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:49.193444 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:49.193616 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:49.194118 22750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:49.195037 22750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:49.195083 22750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.195150 22750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:49.195181 22750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.203491 22750 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:46559
I20260806 23:57:49.203531 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:46559 every 8 connection(s)
I20260806 23:57:49.204319 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:49.210095 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.206056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:32779 (local address 127.8.204.126:44705)
0806 23:57:49.206197 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:49.206201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:49.207137 (+ 936us) server_negotiation.cc:407] Connection header received
0806 23:57:49.208415 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:49.208419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:49.208465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:49.208573 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:49.209675 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:49.209679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:49.209681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:49.209694 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:49.209721 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:49.209729 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:49.209830 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:49.209964 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 23:57:49.209999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:57:49.211002 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.206344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44705 (local address 127.8.204.65:32779)
0806 23:57:49.207000 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:49.207037 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:49.208181 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:49.208728 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:49.208734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:49.208810 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:49.209376 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:49.209378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:49.209443 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:49.209544 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:49.209834 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:49.209843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:49.209848 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:57:49.209984 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 23:57:49.210173 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260806 23:57:49.212204 22922 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44705
I20260806 23:57:49.212393 22922 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:49.212736 22922 heartbeater.cc:511] Master 127.8.204.126:44705 requested a full tablet report, sending...
I20260806 23:57:49.213938 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22750
I20260806 23:57:49.214061 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:49.214147 22669 ts_manager.cc:194] Registered new tserver with Master: 91e4e461027e4ca1838f52810e3e2a60 (127.8.204.65:46559)
I20260806 23:57:49.214442 9009 external_mini_cluster.cc:1428] Setting key e1d5e89eb21feeb656a9a6e94d3707b0
I20260806 23:57:49.216662 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--builtin_ntp_servers=127.8.204.84:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 23:57:49.323065 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:49.323297 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:49.323359 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:49.327180 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:49.327250 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:49.327338 22927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:49.331746 22927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:49.332696 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:49.333896 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:49.342489 22932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:49.342525 22933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:49.342451 22935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:49.342667 22927 server_base.cc:1034] running on GCE node
I20260806 23:57:49.343106 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:49.343665 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:49.344851 22927 hybrid_clock.cc:648] HybridClock initialized: now 1786060669344830 us; error 43 us; skew 500 ppm
I20260806 23:57:49.346479 22927 webserver.cc:460] Webserver started at http://127.8.204.66:42745/ using document root <none> and password file <none>
I20260806 23:57:49.347050 22927 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:49.347132 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:49.347317 22927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:49.349164 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "d679f7798f07470a8d1dcd55c4bf0843"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "3682205bf0259ba25b35c4dedbb57df8"
server_key_iv: "355f3a9a845fba86477d758c7c7aa4ec"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.349701 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "d679f7798f07470a8d1dcd55c4bf0843"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "3682205bf0259ba25b35c4dedbb57df8"
server_key_iv: "355f3a9a845fba86477d758c7c7aa4ec"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.353127 22927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:57:49.355391 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.356420 22927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:49.356539 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "d679f7798f07470a8d1dcd55c4bf0843"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "3682205bf0259ba25b35c4dedbb57df8"
server_key_iv: "355f3a9a845fba86477d758c7c7aa4ec"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.356634 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:49.360720 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:49.361239 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:49.361397 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:49.361867 22927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:49.362639 22927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:49.362741 22927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.362818 22927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:49.362859 22927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.370976 22927 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:34353
I20260806 23:57:49.371024 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:34353 every 8 connection(s)
I20260806 23:57:49.371760 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:49.373104 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 22927
I20260806 23:57:49.373246 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:49.373571 9009 external_mini_cluster.cc:1428] Setting key 1ca80a71da0fb188711feef4f19f57d2
I20260806 23:57:49.375545 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--builtin_ntp_servers=127.8.204.84:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 23:57:49.377583 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.373464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:44059 (local address 127.8.204.126:44705)
0806 23:57:49.373614 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:49.373619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:49.374670 (+ 1051us) server_negotiation.cc:407] Connection header received
0806 23:57:49.376007 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:49.376010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:49.376050 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:49.376126 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:49.377134 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:49.377138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:49.377141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:49.377152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:49.377174 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:49.377183 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:49.377286 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:49.377417 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 23:57:49.377460 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:57:49.378316 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.373854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44705 (local address 127.8.204.66:44059)
0806 23:57:49.374518 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:49.374563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:49.375819 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:49.376267 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:49.376273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:49.376347 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:49.376906 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:49.376907 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:49.376940 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:49.376999 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:49.377275 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:49.377284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:49.377290 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:57:49.377457 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 23:57:49.377665 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":124,"threads_started":1}
I20260806 23:57:49.379348 23084 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44705
I20260806 23:57:49.379590 23084 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:49.380089 23084 heartbeater.cc:511] Master 127.8.204.126:44705 requested a full tablet report, sending...
I20260806 23:57:49.381203 22667 ts_manager.cc:194] Registered new tserver with Master: d679f7798f07470a8d1dcd55c4bf0843 (127.8.204.66:34353)
W20260806 23:57:49.482942 23089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:49.483144 23089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:49.483178 23089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:49.486904 23089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:49.486944 23089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:49.486991 23089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:49.491241 23089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:49.492059 23089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:49.493309 23089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:49.502071 23099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:49.502331 23089 server_base.cc:1034] running on GCE node
W20260806 23:57:49.502071 23098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:49.502081 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:49.502847 23089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:49.503423 23089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:49.504613 23089 hybrid_clock.cc:648] HybridClock initialized: now 1786060669504603 us; error 27 us; skew 500 ppm
I20260806 23:57:49.506342 23089 webserver.cc:460] Webserver started at http://127.8.204.67:38917/ using document root <none> and password file <none>
I20260806 23:57:49.506933 23089 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:49.507019 23089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:49.507237 23089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:49.509207 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "e708613c69034f8dbdffdfd51273e3bb"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "d7e9baaa2ac879b873a8751665dc7584"
server_key_iv: "44f9c9156bcaece9523713657d9ae961"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.509807 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "e708613c69034f8dbdffdfd51273e3bb"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "d7e9baaa2ac879b873a8751665dc7584"
server_key_iv: "44f9c9156bcaece9523713657d9ae961"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.513439 23089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:49.515887 23108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.516935 23089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:49.517066 23089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "e708613c69034f8dbdffdfd51273e3bb"
format_stamp: "Formatted at 2026-08-06 23:57:49 on dist-test-slave-39l8"
server_key: "d7e9baaa2ac879b873a8751665dc7584"
server_key_iv: "44f9c9156bcaece9523713657d9ae961"
server_key_version: "encryptionkey@0"
I20260806 23:57:49.517200 23089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:49.521441 23089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:49.521972 23089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:49.522152 23089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:49.522787 23089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:49.523630 23089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:49.523698 23089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.523761 23089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:49.523810 23089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:49.532361 23089 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:46587
I20260806 23:57:49.532384 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:46587 every 8 connection(s)
I20260806 23:57:49.533574 23089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:49.539265 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.535353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:50441 (local address 127.8.204.126:44705)
0806 23:57:49.535544 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:49.535549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:49.536474 (+ 925us) server_negotiation.cc:407] Connection header received
0806 23:57:49.537643 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:49.537645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:49.537679 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:49.537748 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:49.538799 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:49.538803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:49.538805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:57:49.538855 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:49.538878 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:57:49.538889 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:49.538988 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:49.539132 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 23:57:49.539166 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 23:57:49.540012 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:49.535734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44705 (local address 127.8.204.67:50441)
0806 23:57:49.536329 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:49.536366 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:49.537461 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:49.537888 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:49.537894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:49.537967 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:49.538494 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:57:49.538496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:49.538541 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:57:49.538629 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:49.538989 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:49.539000 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:49.539006 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:57:49.539162 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 23:57:49.539371 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260806 23:57:49.541043 23260 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44705
I20260806 23:57:49.541252 23260 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:49.541610 23260 heartbeater.cc:511] Master 127.8.204.126:44705 requested a full tablet report, sending...
I20260806 23:57:49.542554 22667 ts_manager.cc:194] Registered new tserver with Master: e708613c69034f8dbdffdfd51273e3bb (127.8.204.67:46587)
I20260806 23:57:49.542879 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 23089
I20260806 23:57:49.542991 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:49.543277 9009 external_mini_cluster.cc:1428] Setting key fdc3908000e2539259825f3c4ff65fae
I20260806 23:57:49.545209 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 23:57:49.547793 22719 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:57:49.546983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.8.204.126:44705)
0806 23:57:49.547103 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:49.547107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:49.547315 (+ 208us) server_negotiation.cc:407] Connection header received
0806 23:57:49.547386 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:49.547388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:49.547417 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:49.547479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:49.547689 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49626: BlockingRecv error: recv got EOF from 127.0.0.1:49626 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:57:49.548092 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22750
I20260806 23:57:49.565448 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22927
I20260806 23:57:49.575090 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 23089
I20260806 23:57:49.584828 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 22633
2026-08-06T23:57:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (784 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:49 dist-test-slave-39l8 krb5kdc[23276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:49 dist-test-slave-39l8 krb5kdc[23276](info): set up 2 sockets
Aug 06 23:57:49 dist-test-slave-39l8 krb5kdc[23276](info): commencing operation
krb5kdc: starting...
W20260806 23:57:50.727871 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:50 dist-test-slave-39l8 krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:57:50.845323 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:42173
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:50.951792 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:50.952037 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:50.952103 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:50.955964 23293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:57:50.956029 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:50.956065 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:50.956133 23293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:57:50.956171 23293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:57:50.961104 23293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:42173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:50.961998 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:50.962929 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:50.970763 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:50.970760 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:50.970760 23305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:50.970927 23293 server_base.cc:1034] running on GCE node
I20260806 23:57:50.971544 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:50.972640 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:50.973829 23293 hybrid_clock.cc:648] HybridClock initialized: now 1786060670973802 us; error 38 us; skew 500 ppm
Aug 06 23:57:50 dist-test-slave-39l8 krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060670, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:50.976444 23293 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:57:50.977228 23293 webserver.cc:460] Webserver started at http://127.8.204.126:46771/ using document root <none> and password file <none>
I20260806 23:57:50.977774 23293 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:50.977864 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:50.978087 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:50.980094 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "4ebeae5db42d41d0aca81bed2ecb4330"
format_stamp: "Formatted at 2026-08-06 23:57:50 on dist-test-slave-39l8"
server_key: "85d77292b3b4ed8d06f8d8a107c80c79"
server_key_iv: "57ca09e6c1e201d066b408c19f1650cc"
server_key_version: "encryptionkey@0"
I20260806 23:57:50.980681 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "4ebeae5db42d41d0aca81bed2ecb4330"
format_stamp: "Formatted at 2026-08-06 23:57:50 on dist-test-slave-39l8"
server_key: "85d77292b3b4ed8d06f8d8a107c80c79"
server_key_iv: "57ca09e6c1e201d066b408c19f1650cc"
server_key_version: "encryptionkey@0"
I20260806 23:57:50.984032 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:57:50.986332 23316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:50.987502 23293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:50.987648 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "4ebeae5db42d41d0aca81bed2ecb4330"
format_stamp: "Formatted at 2026-08-06 23:57:50 on dist-test-slave-39l8"
server_key: "85d77292b3b4ed8d06f8d8a107c80c79"
server_key_iv: "57ca09e6c1e201d066b408c19f1650cc"
server_key_version: "encryptionkey@0"
I20260806 23:57:50.987751 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:51.005128 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:51.007746 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:51.007929 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:51.014635 23293 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:42173
I20260806 23:57:51.014958 23377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:42173 every 8 connection(s)
I20260806 23:57:51.015632 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:57:51.018378 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:51.020818 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 23293
I20260806 23:57:51.020957 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:57:51.021235 9009 external_mini_cluster.cc:1428] Setting key affd58b8999ec7a72cd2f28b2de22653
I20260806 23:57:51.024545 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: Bootstrap starting.
I20260806 23:57:51.027218 23380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:51.028036 23380 log.cc:824] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:51.029996 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: No bootstrap required, opened a new log
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:51.033617 23380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } }
I20260806 23:57:51.033877 23380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:51.033955 23380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebeae5db42d41d0aca81bed2ecb4330, State: Initialized, Role: FOLLOWER
I20260806 23:57:51.034440 23380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } }
I20260806 23:57:51.034607 23380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:57:51.034698 23380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:57:51.034849 23380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:51.035962 23380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } }
I20260806 23:57:51.036321 23380 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebeae5db42d41d0aca81bed2ecb4330; no voters:
I20260806 23:57:51.036684 23380 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:57:51.036756 23386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:51.036958 23386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [term 1 LEADER]: Becoming Leader. State: Replica: 4ebeae5db42d41d0aca81bed2ecb4330, State: Running, Role: LEADER
I20260806 23:57:51.036955 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.022658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.8.204.126:42173)
0806 23:57:51.023129 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.023149 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.023195 (+ 46us) server_negotiation.cc:407] Connection header received
0806 23:57:51.024005 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.024026 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.024425 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.024793 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:51.025454 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.026955 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.027860 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.028527 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.028642 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.031394 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:51.031399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:51.031411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:51.031438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:51.033700 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.034235 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.034241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.034247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.034376 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.034726 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.034730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.034732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.035215 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:51.035371 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.035653 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.036097 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":98,"threads_started":1}
I20260806 23:57:51.037272 23386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } }
I20260806 23:57:51.037937 23380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:57:51.039662 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebeae5db42d41d0aca81bed2ecb4330. Latest consensus state: current_term: 1 leader_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } } }
I20260806 23:57:51.039954 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [sys.catalog]: This master's current role is: LEADER
I20260806 23:57:51.040346 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebeae5db42d41d0aca81bed2ecb4330" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 42173 } } }
I20260806 23:57:51.040442 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330 [sys.catalog]: This master's current role is: LEADER
W20260806 23:57:51.042765 23414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:57:51.042837 23414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:57:51.042914 23411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:57:51.043601 23411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:57:51.048800 23411 catalog_manager.cc:1347] Generated new cluster ID: 7e4570abc2d6496caf72794259cac0fe
I20260806 23:57:51.048870 23411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:57:51.093276 23411 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:57:51.094529 23411 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:57:51.106202 23411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: Generated new TSK 0
I20260806 23:57:51.107012 23411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:57:51.143239 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:51.248107 23420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:51.248318 23420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:51.248374 23420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:51.252220 23420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:51.252267 23420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:51.252348 23420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:57:51.257156 23420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:51.258034 23420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:51.259018 23420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:51.268047 23427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:51.268111 23420 server_base.cc:1034] running on GCE node
W20260806 23:57:51.268093 23429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:51.268047 23426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:51.268684 23420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:51.269176 23420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:51.270334 23420 hybrid_clock.cc:648] HybridClock initialized: now 1786060671270318 us; error 28 us; skew 500 ppm
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:51.272992 23420 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:57:51.275143 23420 webserver.cc:460] Webserver started at http://127.8.204.65:41081/ using document root <none> and password file <none>
I20260806 23:57:51.275681 23420 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:51.275770 23420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:51.275990 23420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:51.277801 23420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "7c598d2652e840b2be9ea90f237a52f2"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "344ea08fc51f4d870af686573b024a5a"
server_key_iv: "3c90993a0764058a4f350bb70b97a527"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.278334 23420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "7c598d2652e840b2be9ea90f237a52f2"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "344ea08fc51f4d870af686573b024a5a"
server_key_iv: "3c90993a0764058a4f350bb70b97a527"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.281664 23420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:57:51.283934 23439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.285053 23420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:57:51.285195 23420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "7c598d2652e840b2be9ea90f237a52f2"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "344ea08fc51f4d870af686573b024a5a"
server_key_iv: "3c90993a0764058a4f350bb70b97a527"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.285300 23420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:51.299810 23420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:51.302630 23420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:51.302850 23420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:51.303390 23420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:51.304105 23420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:51.304170 23420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.304258 23420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:51.304306 23420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.312673 23420 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:34521
I20260806 23:57:51.312695 23589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:34521 every 8 connection(s)
I20260806 23:57:51.313433 23420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:57:51.319574 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 23420
I20260806 23:57:51.319701 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:57:51.319993 9009 external_mini_cluster.cc:1428] Setting key 1e648aa5ef3567ad20dcac7d11286070
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:51.327064 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.315149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46181 (local address 127.8.204.126:42173)
0806 23:57:51.315282 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.315286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.315969 (+ 683us) server_negotiation.cc:407] Connection header received
0806 23:57:51.317235 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.317238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.317300 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.317404 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:51.318431 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.319076 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.320345 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.320637 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.320742 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.323329 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:51.323336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:51.323342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:51.323376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:51.324845 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.325463 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.325466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.325467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.325552 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.325940 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.325942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.325944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.326093 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:51.326182 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.326847 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.326949 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:57:51.328192 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.315399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42173 (local address 127.8.204.65:46181)
0806 23:57:51.315827 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.315864 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.317024 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.317568 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.317574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.317883 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:51.318289 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.318299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.319207 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.319213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.320230 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.320240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.320817 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.320821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.320994 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.321559 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:51.321574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:51.323155 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:51.324968 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.324978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.324994 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.325361 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.325642 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.325648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.325653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.325851 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.326214 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:51.326223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:51.326555 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.326799 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.327147 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260806 23:57:51.329413 23591 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42173
I20260806 23:57:51.329679 23591 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:51.330241 23591 heartbeater.cc:511] Master 127.8.204.126:42173 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:57:51.331849 23335 ts_manager.cc:194] Registered new tserver with Master: 7c598d2652e840b2be9ea90f237a52f2 (127.8.204.65:34521)
I20260806 23:57:51.333022 23335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:46181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:57:51.347975 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:51.452790 23598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:51.453006 23598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:51.453045 23598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:51.456763 23598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:51.456808 23598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:51.456875 23598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:57:51.461352 23598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:51.462170 23598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:51.463111 23598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:51.471765 23609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:51.471837 23598 server_base.cc:1034] running on GCE node
W20260806 23:57:51.471791 23612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:51.471819 23610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:51.472441 23598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:51.472955 23598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:51.474116 23598 hybrid_clock.cc:648] HybridClock initialized: now 1786060671474101 us; error 27 us; skew 500 ppm
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:51.476753 23598 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:57:51.477573 23598 webserver.cc:460] Webserver started at http://127.8.204.66:40877/ using document root <none> and password file <none>
I20260806 23:57:51.478087 23598 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:51.478173 23598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:51.478397 23598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:51.480229 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "a51e8f106ae943da85253f5cd167b646"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "6add9a1d0fcefb231b36e234cc7f72fb"
server_key_iv: "4339376f33a8facfd93af68c458a74dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.480775 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "a51e8f106ae943da85253f5cd167b646"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "6add9a1d0fcefb231b36e234cc7f72fb"
server_key_iv: "4339376f33a8facfd93af68c458a74dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.484190 23598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:57:51.486409 23621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.487509 23598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:57:51.487637 23598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "a51e8f106ae943da85253f5cd167b646"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "6add9a1d0fcefb231b36e234cc7f72fb"
server_key_iv: "4339376f33a8facfd93af68c458a74dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.487744 23598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:51.508155 23598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:51.510473 23598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:51.510643 23598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:51.511200 23598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:51.511957 23598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:51.512095 23598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.512176 23598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:51.512225 23598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.520562 23598 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:45909
I20260806 23:57:51.520596 23785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:45909 every 8 connection(s)
I20260806 23:57:51.521337 23598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:57:51.523897 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 23598
I20260806 23:57:51.524010 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:57:51.524271 9009 external_mini_cluster.cc:1428] Setting key 40f7b03725e4d109311cc81ee65558d1
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:57:51.535713 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.522977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:33083 (local address 127.8.204.126:42173)
0806 23:57:51.523115 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.523118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.523875 (+ 757us) server_negotiation.cc:407] Connection header received
0806 23:57:51.525386 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.525389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.525447 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.525556 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:51.527068 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.527552 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.528785 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.529023 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.529144 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.531925 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:51.531928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:51.531930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:51.531952 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:51.533414 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.534034 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.534036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.534037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.534116 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.534552 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.534554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.534555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.534780 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:51.534858 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.535477 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.535592 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:57:51.536615 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.523321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42173 (local address 127.8.204.66:33083)
0806 23:57:51.523737 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.523775 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.525181 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.525771 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.525780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.526239 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:51.526905 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.526927 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.527692 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.527698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.528653 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.528664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.529179 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.529183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.529343 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.529902 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:51.529918 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:51.531747 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:51.533529 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.533538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.533554 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.533934 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.534222 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.534225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.534227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.534442 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.534885 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:51.534891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:51.535213 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.535426 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.535675 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260806 23:57:51.537833 23786 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42173
I20260806 23:57:51.538094 23786 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:51.538538 23786 heartbeater.cc:511] Master 127.8.204.126:42173 requested a full tablet report, sending...
I20260806 23:57:51.539655 23335 ts_manager.cc:194] Registered new tserver with Master: a51e8f106ae943da85253f5cd167b646 (127.8.204.66:45909)
I20260806 23:57:51.540375 23335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:33083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:57:51.551813 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:57:51.657274 23791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:57:51.657490 23791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:57:51.657555 23791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:57:51.661520 23791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:57:51.661568 23791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:57:51.661620 23791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:57:51.666402 23791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:57:51.667364 23791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:57:51.668269 23791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:57:51.677083 23802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:51.677161 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:57:51.677083 23800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:57:51.677366 23791 server_base.cc:1034] running on GCE node
I20260806 23:57:51.677780 23791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:57:51.678333 23791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:57:51.679512 23791 hybrid_clock.cc:648] HybridClock initialized: now 1786060671679498 us; error 25 us; skew 500 ppm
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:57:51.687161 23791 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:57:51.688046 23791 webserver.cc:460] Webserver started at http://127.8.204.67:33135/ using document root <none> and password file <none>
I20260806 23:57:51.688596 23791 fs_manager.cc:359] Metadata directory not provided
I20260806 23:57:51.688683 23791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:57:51.688918 23791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:57:51.690999 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "315f81d1eba04fc5abc6b2dff3d36b96"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "b68c0048e692f079b84f8cbce217fe40"
server_key_iv: "24fdc304ff168acec5065d0655b273dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.691550 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "315f81d1eba04fc5abc6b2dff3d36b96"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "b68c0048e692f079b84f8cbce217fe40"
server_key_iv: "24fdc304ff168acec5065d0655b273dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.695051 23791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:57:51.697239 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.698401 23791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:57:51.698511 23791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "315f81d1eba04fc5abc6b2dff3d36b96"
format_stamp: "Formatted at 2026-08-06 23:57:51 on dist-test-slave-39l8"
server_key: "b68c0048e692f079b84f8cbce217fe40"
server_key_iv: "24fdc304ff168acec5065d0655b273dc"
server_key_version: "encryptionkey@0"
I20260806 23:57:51.698603 23791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:57:51.708702 23791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:57:51.711572 23791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:57:51.711752 23791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:57:51.712296 23791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:57:51.713038 23791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:57:51.713083 23791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.713147 23791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:57:51.713193 23791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:57:51.721498 23791 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:41723
I20260806 23:57:51.721544 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:41723 every 8 connection(s)
I20260806 23:57:51.722282 23791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:57:51.727880 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 23791
I20260806 23:57:51.728019 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:57:51.728295 9009 external_mini_cluster.cc:1428] Setting key 9ca62a62ccb8da539265a696c83dd46a
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:57:51.735047 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.723998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:54135 (local address 127.8.204.126:42173)
0806 23:57:51.724139 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.724142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.725078 (+ 936us) server_negotiation.cc:407] Connection header received
0806 23:57:51.725867 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.725869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.725914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.725988 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:51.727064 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.727563 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.728736 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.728992 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.729100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.731620 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:51.731623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:51.731624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:51.731649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:51.733060 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.733561 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.733565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.733567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.733649 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.733998 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.734001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.734003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.734173 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:51.734281 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.734767 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.734884 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:57:51.735734 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.724313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:42173 (local address 127.8.204.67:54135)
0806 23:57:51.724942 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.724979 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.725710 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.726141 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.726147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.726466 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:57:51.726945 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.726958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.727666 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.727669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.728609 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.728619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.729149 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.729153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.729354 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.729953 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:57:51.729968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:57:51.731453 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:57:51.733197 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.733203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.733216 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.733459 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.733761 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:57:51.733765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:57:51.733767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:57:51.733899 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:57:51.734280 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:57:51.734287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:57:51.734523 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.734700 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.735003 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":104,"threads_started":1}
I20260806 23:57:51.736646 23956 heartbeater.cc:347] Connected to a master server at 127.8.204.126:42173
I20260806 23:57:51.736828 23956 heartbeater.cc:464] Registering TS with master...
I20260806 23:57:51.737267 23956 heartbeater.cc:511] Master 127.8.204.126:42173 requested a full tablet report, sending...
I20260806 23:57:51.738175 23335 ts_manager.cc:194] Registered new tserver with Master: 315f81d1eba04fc5abc6b2dff3d36b96 (127.8.204.67:41723)
I20260806 23:57:51.738780 23335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:54135
I20260806 23:57:51.741925 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:57:51.750027 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.743703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.8.204.126:42173)
0806 23:57:51.743805 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.743808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.743955 (+ 147us) server_negotiation.cc:407] Connection header received
0806 23:57:51.744040 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.744041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.744073 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.744134 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:57:51.744764 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.745215 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.745953 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.746129 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.746197 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.747377 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:57:51.747379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:57:51.747381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:57:51.747402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:57:51.748729 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.749067 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.749070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.749071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.749151 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:57:51.749361 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:57:51.749363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:57:51.749364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:57:51.749556 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:57:51.749661 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.749860 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.749936 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:57:51.752446 23335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:57:51.755241 23335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:57:51.768263 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.762654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:34521 (local address 127.0.0.1:33316)
0806 23:57:51.763393 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.763438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.763640 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.764404 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.764408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.764428 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:51.764519 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.764524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.765741 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.765745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.766668 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.766678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.767722 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.767725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.767851 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.767895 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.768006 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.768125 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260806 23:57:51.768271 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.762795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.8.204.65:34521)
0806 23:57:51.763273 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.763285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.763529 (+ 244us) server_negotiation.cc:407] Connection header received
0806 23:57:51.763840 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.763846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.764051 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.764167 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:51.764636 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.765609 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.766795 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.767588 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.767732 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.767895 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.768032 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.768135 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":110,"threads_started":1}
I20260806 23:57:51.769181 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.763128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41723 (local address 127.0.0.1:57046)
0806 23:57:51.763583 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.763597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.763673 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.765128 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.765131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.765144 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:51.765215 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.765220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.766499 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.766505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.767676 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.767689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.768814 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.768817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.768901 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.768921 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.769004 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.769068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260806 23:57:51.769232 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.763277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.8.204.67:41723)
0806 23:57:51.764644 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.764655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.764675 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:57:51.764736 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.764741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.764898 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.765031 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:51.765337 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.766340 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.767798 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.768681 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.768815 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.768937 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.769030 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.769107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":131,"threads_started":1}
I20260806 23:57:51.770515 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.763315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45909 (local address 127.0.0.1:36780)
0806 23:57:51.764290 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.764301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.764414 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.764896 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.764900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.764920 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:51.764995 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.765001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.766443 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.766446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.768944 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.768955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.770139 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.770143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.770227 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.770248 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.770327 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.770385 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":1196,"thread_start_us":88,"threads_started":1}
I20260806 23:57:51.771034 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.763758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.8.204.66:45909)
0806 23:57:51.764083 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.764097 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.764405 (+ 308us) server_negotiation.cc:407] Connection header received
0806 23:57:51.764532 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.764537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.764737 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.764893 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:51.765097 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.766193 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.769080 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.769838 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.769940 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.770062 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.770805 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.770899 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":130,"threads_started":1}
I20260806 23:57:51.771652 23505 tablet_service.cc:1467] Processing CreateTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c (DEFAULT_TABLE table=test-table [id=997da3d41e1a4f659b093dc53cd9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:51.771942 23878 tablet_service.cc:1467] Processing CreateTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c (DEFAULT_TABLE table=test-table [id=997da3d41e1a4f659b093dc53cd9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:51.772722 23505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55bc46968ec4f2e9f40d5abaa4a965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:51.772926 23878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55bc46968ec4f2e9f40d5abaa4a965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:51.773479 23690 tablet_service.cc:1467] Processing CreateTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c (DEFAULT_TABLE table=test-table [id=997da3d41e1a4f659b093dc53cd9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:57:51.774322 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55bc46968ec4f2e9f40d5abaa4a965c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:57:51.779131 23980 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Bootstrap starting.
I20260806 23:57:51.780071 23984 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Bootstrap starting.
I20260806 23:57:51.780861 23985 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Bootstrap starting.
I20260806 23:57:51.781867 23980 tablet_bootstrap.cc:654] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:51.782244 23984 tablet_bootstrap.cc:654] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:51.782691 23980 log.cc:824] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:51.782968 23984 log.cc:824] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:51.783298 23985 tablet_bootstrap.cc:654] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Neither blocks nor log segments found. Creating new log.
I20260806 23:57:51.784109 23985 log.cc:824] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Log is configured to *not* fsync() on all Append() calls
I20260806 23:57:51.784870 23980 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: No bootstrap required, opened a new log
I20260806 23:57:51.784968 23984 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: No bootstrap required, opened a new log
I20260806 23:57:51.785097 23980 ts_tablet_manager.cc:1397] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 23:57:51.785181 23984 ts_tablet_manager.cc:1397] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:57:51.786103 23985 tablet_bootstrap.cc:492] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: No bootstrap required, opened a new log
I20260806 23:57:51.786340 23985 ts_tablet_manager.cc:1397] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 23:57:51.788282 23984 raft_consensus.cc:348] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.788539 23984 raft_consensus.cc:374] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:51.788625 23984 raft_consensus.cc:729] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f81d1eba04fc5abc6b2dff3d36b96, State: Initialized, Role: FOLLOWER
I20260806 23:57:51.788897 23980 raft_consensus.cc:348] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.789165 23980 raft_consensus.cc:374] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:51.789075 23984 consensus_queue.cc:260] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.789253 23980 raft_consensus.cc:729] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c598d2652e840b2be9ea90f237a52f2, State: Initialized, Role: FOLLOWER
I20260806 23:57:51.789764 23956 heartbeater.cc:503] Master 127.8.204.126:42173 was elected leader, sending a full tablet report...
I20260806 23:57:51.790023 23984 ts_tablet_manager.cc:1428] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:57:51.790076 23985 raft_consensus.cc:348] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.790315 23985 raft_consensus.cc:374] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:57:51.790359 23985 raft_consensus.cc:729] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51e8f106ae943da85253f5cd167b646, State: Initialized, Role: FOLLOWER
I20260806 23:57:51.790870 23980 consensus_queue.cc:260] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.790849 23985 consensus_queue.cc:260] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.791862 23985 ts_tablet_manager.cc:1428] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:57:51.791880 23980 ts_tablet_manager.cc:1428] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 23:57:51.792219 23591 heartbeater.cc:503] Master 127.8.204.126:42173 was elected leader, sending a full tablet report...
I20260806 23:57:51.792255 23786 heartbeater.cc:503] Master 127.8.204.126:42173 was elected leader, sending a full tablet report...
W20260806 23:57:51.814527 23592 tablet.cc:2369] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:57:51.839952 23991 raft_consensus.cc:482] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:57:51.840169 23991 raft_consensus.cc:504] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.841331 23991 leader_election.cc:288] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51e8f106ae943da85253f5cd167b646 (127.8.204.66:45909), 315f81d1eba04fc5abc6b2dff3d36b96 (127.8.204.67:41723)
I20260806 23:57:51.845257 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.841779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41325 (local address 127.8.204.66:45909)
0806 23:57:51.841895 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.841900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.842113 (+ 213us) server_negotiation.cc:407] Connection header received
0806 23:57:51.842192 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.842195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.842264 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.842342 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:51.842692 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.843143 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844265 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.844665 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844793 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.844846 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.845050 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.845120 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:57:51.845263 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.841664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:40621 (local address 127.8.204.67:41723)
0806 23:57:51.841798 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.841804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.841977 (+ 173us) server_negotiation.cc:407] Connection header received
0806 23:57:51.842100 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.842104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.842186 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.842258 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:57:51.842566 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.843093 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844263 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.844666 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844793 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.844853 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.845069 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.845145 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:57:51.845278 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.841556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:45909 (local address 127.8.204.65:41325)
0806 23:57:51.841993 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.842008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.842114 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.842442 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.842444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.842457 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:51.842560 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.842567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.843274 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.843279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.844138 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.844148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.844787 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.844913 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.844943 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.845031 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.845101 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":43,"thread_start_us":103,"threads_started":1}
I20260806 23:57:51.845283 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.841556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:41723 (local address 127.8.204.65:40621)
0806 23:57:51.841908 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.841921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:57:51.842007 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:57:51.842310 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:57:51.842315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:57:51.842336 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:57:51.842458 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.842464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.843200 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.843204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.844142 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:57:51.844152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.844791 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.844795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:57:51.844944 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.844962 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:57:51.845051 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 23:57:51.845120 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":115,"thread_start_us":104,"threads_started":1}
I20260806 23:57:51.845942 23714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55bc46968ec4f2e9f40d5abaa4a965c" candidate_uuid: "7c598d2652e840b2be9ea90f237a52f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51e8f106ae943da85253f5cd167b646" is_pre_election: true
I20260806 23:57:51.845993 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55bc46968ec4f2e9f40d5abaa4a965c" candidate_uuid: "7c598d2652e840b2be9ea90f237a52f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" is_pre_election: true
I20260806 23:57:51.846251 23714 raft_consensus.cc:2454] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c598d2652e840b2be9ea90f237a52f2 in term 0.
I20260806 23:57:51.846259 23902 raft_consensus.cc:2454] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c598d2652e840b2be9ea90f237a52f2 in term 0.
I20260806 23:57:51.846745 23440 leader_election.cc:302] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c598d2652e840b2be9ea90f237a52f2, a51e8f106ae943da85253f5cd167b646; no voters:
I20260806 23:57:51.847035 23991 raft_consensus.cc:2781] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:57:51.847128 23991 raft_consensus.cc:482] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:57:51.847160 23991 raft_consensus.cc:3037] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:51.848188 23991 raft_consensus.cc:504] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.848537 23991 leader_election.cc:288] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [CANDIDATE]: Term 1 election: Requested vote from peers a51e8f106ae943da85253f5cd167b646 (127.8.204.66:45909), 315f81d1eba04fc5abc6b2dff3d36b96 (127.8.204.67:41723)
I20260806 23:57:51.848881 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55bc46968ec4f2e9f40d5abaa4a965c" candidate_uuid: "7c598d2652e840b2be9ea90f237a52f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f81d1eba04fc5abc6b2dff3d36b96"
I20260806 23:57:51.848896 23714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55bc46968ec4f2e9f40d5abaa4a965c" candidate_uuid: "7c598d2652e840b2be9ea90f237a52f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51e8f106ae943da85253f5cd167b646"
I20260806 23:57:51.848997 23714 raft_consensus.cc:3037] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:51.848999 23902 raft_consensus.cc:3037] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:57:51.850246 23714 raft_consensus.cc:2454] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c598d2652e840b2be9ea90f237a52f2 in term 1.
I20260806 23:57:51.850246 23902 raft_consensus.cc:2454] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c598d2652e840b2be9ea90f237a52f2 in term 1.
I20260806 23:57:51.850525 23440 leader_election.cc:302] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c598d2652e840b2be9ea90f237a52f2, a51e8f106ae943da85253f5cd167b646; no voters:
I20260806 23:57:51.850770 23991 raft_consensus.cc:2781] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:57:51.850955 23991 raft_consensus.cc:686] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [term 1 LEADER]: Becoming Leader. State: Replica: 7c598d2652e840b2be9ea90f237a52f2, State: Running, Role: LEADER
I20260806 23:57:51.851280 23991 consensus_queue.cc:237] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } }
I20260806 23:57:51.854326 23333 catalog_manager.cc:5495] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c598d2652e840b2be9ea90f237a52f2 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "7c598d2652e840b2be9ea90f237a52f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c598d2652e840b2be9ea90f237a52f2" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 } health_report { overall_health: UNKNOWN } } }
I20260806 23:57:51.866072 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.863066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.8.204.65:34521)
0806 23:57:51.863176 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 23:57:51.863180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:57:51.863229 (+ 49us) server_negotiation.cc:407] Connection header received
0806 23:57:51.863344 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:57:51.863346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:57:51.863390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:57:51.863467 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:57:51.863661 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.864063 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.864832 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:57:51.865101 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:57:51.865202 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:57:51.865441 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:57:51.865818 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:57:51.865898 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:57:51.865948 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 23:57:51.865988 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:57:51.871321 23902 raft_consensus.cc:1261] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 1 FOLLOWER]: Refusing update from remote peer 7c598d2652e840b2be9ea90f237a52f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:51.871320 23714 raft_consensus.cc:1261] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 1 FOLLOWER]: Refusing update from remote peer 7c598d2652e840b2be9ea90f237a52f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:57:51.872008 23991 consensus_queue.cc:1035] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51e8f106ae943da85253f5cd167b646" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:51.872257 23996 consensus_queue.cc:1035] T d55bc46968ec4f2e9f40d5abaa4a965c P 7c598d2652e840b2be9ea90f237a52f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f81d1eba04fc5abc6b2dff3d36b96" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:57:51.881404 24004 mvcc.cc:205] Tried to move back new op lower bound from 7315704511978422272 to 7315704511905505280. Current Snapshot: MvccSnapshot[applied={T|T < 7315704511978422272}]
I20260806 23:57:51.881481 24006 mvcc.cc:205] Tried to move back new op lower bound from 7315704511978422272 to 7315704511905505280. Current Snapshot: MvccSnapshot[applied={T|T < 7315704511978422272}]
I20260806 23:57:51.886528 23333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:51.886698 23333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:57:51.888896 23333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebeae5db42d41d0aca81bed2ecb4330: Sending DeleteTablet for 3 replicas of tablet d55bc46968ec4f2e9f40d5abaa4a965c
I20260806 23:57:51.889752 23690 tablet_service.cc:1514] Processing DeleteTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:51 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:36780
I20260806 23:57:51.889745 23505 tablet_service.cc:1514] Processing DeleteTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:51 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260806 23:57:51.889779 23878 tablet_service.cc:1514] Processing DeleteTablet for tablet d55bc46968ec4f2e9f40d5abaa4a965c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:57:51 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:57046
I20260806 23:57:51.890344 24019 tablet_replica.cc:330] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: stopping tablet replica
I20260806 23:57:51.890733 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 23420
I20260806 23:57:51.890776 24019 raft_consensus.cc:2229] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:51.896636 24019 raft_consensus.cc:2258] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:51.896991 24016 tablet_replica.cc:330] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: stopping tablet replica
I20260806 23:57:51.897316 24016 raft_consensus.cc:2229] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:57:51.897575 24016 raft_consensus.cc:2258] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:57:51.898234 24019 ts_tablet_manager.cc:1905] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:51.899618 24016 ts_tablet_manager.cc:1905] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:57:51.900524 24019 ts_tablet_manager.cc:1918] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:51.900652 24019 log.cc:1196] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/d55bc46968ec4f2e9f40d5abaa4a965c
I20260806 23:57:51.900982 24019 ts_tablet_manager.cc:1939] T d55bc46968ec4f2e9f40d5abaa4a965c P a51e8f106ae943da85253f5cd167b646: Deleting consensus metadata
I20260806 23:57:51.902141 23317 catalog_manager.cc:4841] TS a51e8f106ae943da85253f5cd167b646 (127.8.204.66:45909): tablet d55bc46968ec4f2e9f40d5abaa4a965c (table test-table [id=997da3d41e1a4f659b093dc53cd9d388]) successfully deleted
I20260806 23:57:51.903316 24016 ts_tablet_manager.cc:1918] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:57:51.903424 24016 log.cc:1196] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/d55bc46968ec4f2e9f40d5abaa4a965c
I20260806 23:57:51.903754 24016 ts_tablet_manager.cc:1939] T d55bc46968ec4f2e9f40d5abaa4a965c P 315f81d1eba04fc5abc6b2dff3d36b96: Deleting consensus metadata
I20260806 23:57:51.904898 23319 catalog_manager.cc:4841] TS 315f81d1eba04fc5abc6b2dff3d36b96 (127.8.204.67:41723): tablet d55bc46968ec4f2e9f40d5abaa4a965c (table test-table [id=997da3d41e1a4f659b093dc53cd9d388]) successfully deleted
W20260806 23:57:51.907797 23317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:34521 (error 108)
I20260806 23:57:51.907954 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 23598
I20260806 23:57:51.908960 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:57:51.908647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:34521 (local address 127.0.0.1:33330)
0806 23:57:51.908811 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 23:57:51.908874 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 23:57:51.909101 23317 catalog_manager.cc:4584] TS 7c598d2652e840b2be9ea90f237a52f2 (127.8.204.65:34521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d55bc46968ec4f2e9f40d5abaa4a965c: Network error: Client connection negotiation failed: client connection to 127.8.204.65:34521: connect: Connection refused (error 111)
I20260806 23:57:51.918087 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 23791
I20260806 23:57:51.928591 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 23293
2026-08-06T23:57:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2347 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[24026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[24026](info): set up 2 sockets
Aug 06 23:57:51 dist-test-slave-39l8 krb5kdc[24026](info): commencing operation
krb5kdc: starting...
W20260806 23:57:53.076150 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:57:53 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:57:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangeradmin_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangeradmin_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangerlookup_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangerlookup_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
I20260806 23:57:53.235445 9009 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcJxSMk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/postgres -l logfile start
2026-08-06 23:57:55.402 UTC [24061] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 23:57:55.402 UTC [24061] LOG: listening on IPv4 address "127.8.204.84", port 50735
2026-08-06 23:57:55.405 UTC [24061] LOG: listening on Unix socket "/tmp/.s.PGSQL.50735"
2026-08-06 23:57:55.424 UTC [24066] LOG: database system was shut down at 2026-08-06 23:57:54 UTC
2026-08-06 23:57:55.428 UTC [24061] LOG: database system is ready to accept connections
I20260806 23:57:56.468618 9009 mini_postgres.cc:96] Postgres bound to 50735
2026-08-06 23:57:56.476 UTC [24076] FATAL: database "slave" does not exist
127.8.204.84:50735 - accepting connections
I20260806 23:57:56.477507 9009 mini_ranger.cc:161] Starting Ranger...
I20260806 23:57:56.492815 9009 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 23:57:56.884526 9009 mini_ranger.cc:87] Created ranger Postgres database
I20260806 23:57:56.884634 9009 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin
2026-08-06 23:57:57,156 [I] DB FLAVOR :POSTGRES
2026-08-06 23:57:57,156 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:57:57,156 [I] Checking connection..
2026-08-06 23:57:57,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 23:57:57,405 [I] Checking connection passed.
2026-08-06 23:57:57,405 [I] --------- Verifying version history table ---------
2026-08-06 23:57:57,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:57:57,667 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:57:57,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:57:57,915 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:57:57,916 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 23:57:57,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 23:57:58.138 UTC [24160] WARNING: there is no transaction in progress
2026-08-06 23:57:58,146 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 23:57:58,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:57:58,400 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 23:57:58,401 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:57:58,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 23:57:58,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 23:57:58,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-39l8.c.gcp-upstream.internal','N') ;"
2026-08-06 23:57:59,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:57:59,402 [I] Table x_portal_user does not exist in database ranger
2026-08-06 23:57:59,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:57:59,635 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 23:57:59,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:57:59,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:58:00,117 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 23:58:00,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 23:58:00.972 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:00.987 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.001 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.015 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.029 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.045 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.103 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.109 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.123 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.135 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.144 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.156 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.165 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.170 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.176 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01.182 UTC [24303] WARNING: there is no transaction in progress
2026-08-06 23:58:01,736 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 23:58:01,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 23:58:01,982 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 23:58:01,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:58:02,242 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 23:58:02,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:58:02,484 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 23:58:02,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:58:02,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:58:02,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-39l8.c.gcp-upstream.internal';"
2026-08-06 23:58:03,232 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 23:58:03,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 23:58:03,454 [I] DB_PATCHES have already been applied
I20260806 23:58:03.460044 9009 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcJxSMk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 23:58:03.460103 9009 mini_ranger.cc:193] Using host: 127.8.204.84
I20260806 23:58:03.464619 9009 mini_ranger.cc:227] Ranger admin URL: http://127.8.204.84:45587
Aug 06, 2026 11:58:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:58:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:58:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:58:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:58:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 23:58:04.697461 9009 mini_ranger.cc:160] Time spent starting Ranger: real 8.220s user 0.001s sys 0.010s
Aug 06, 2026 11:58:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:58:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 11:58:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:58:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:58:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 23:58:25.146415 9009 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangerkms_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/rangerkms_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 23:58:25 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060705, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 23:58:25.240253 9009 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 23:58:25.250790 9009 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 23:58:25.982795 9009 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 23:58:25.982872 9009 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms
I20260806 23:58:25.982964 9009 mini_ranger_kms.cc:227] Using postgres at 127.8.204.84:50735
2026-08-06 23:58:26,288 [I] DB FLAVOR :POSTGRES
2026-08-06 23:58:26,288 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:58:26,288 [I] Checking connection
2026-08-06 23:58:26,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 23:58:26,513 [I] connection success
2026-08-06 23:58:26,513 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 23:58:26,513 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 23:58:26,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 23:58:26,733 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 23:58:26,733 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:58:26,734 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 23:58:26,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:50735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 23:58:26,971 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 23:58:27.414616 9009 mini_ranger_kms.cc:310] Created kms service
I20260806 23:58:27.579499 9009 mini_ranger_kms.cc:326] Created kudu user
I20260806 23:58:27.623085 9009 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 23:58:27 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060705, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:58:27.992937 9009 mini_ranger_kms.cc:384] Added ranger policy
I20260806 23:58:27.993090 9009 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 23:58:27.993112 9009 mini_ranger_kms.cc:240] Using host: 127.8.204.84
I20260806 23:58:27.998502 9009 mini_ranger_kms.cc:276] Ranger KMS PID: 24551
I20260806 23:58:27.998620 9009 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.204.84:47061
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:58:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 23:58:30 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:58:30.573053 9009 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.333s user 0.000s sys 0.286s
I20260806 23:58:30.573185 9009 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 23:58:30.573315 9009 mini_ranger_kms.cc:401] 127.8.204.84:47061/kms/v1/keys
Aug 06 23:58:30 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:58:31 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:58:33 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:58:33.278501 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41809
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:41809
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:47061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:58:33.407538 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:58:33.407747 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:58:33.407781 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:58:33.411557 24616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:58:33.411607 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:58:33.411624 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:58:33.411638 24616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:58:33.411652 24616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:58:33.417441 24616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.204.84:47061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:41809
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:58:33.418416 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:58:33.419448 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:58:33.428114 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:33.428174 24625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:33.428169 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:33.428612 24616 server_base.cc:1034] running on GCE node
I20260806 23:58:33.429230 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:58:33.430397 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:58:33.431577 24616 hybrid_clock.cc:648] HybridClock initialized: now 1786060713431548 us; error 48 us; skew 500 ppm
Aug 06 23:58:33 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:58:33.440052 24616 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:58:33.440913 24616 webserver.cc:460] Webserver started at http://127.8.204.126:32791/ using document root <none> and password file <none>
I20260806 23:58:33.441432 24616 fs_manager.cc:359] Metadata directory not provided
I20260806 23:58:33.441510 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:58:33.441764 24616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:58:33 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:58:33.502292 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "bc98d3454e0549faafaf9caf085e06ce"
format_stamp: "Formatted at 2026-08-06 23:58:33 on dist-test-slave-39l8"
server_key: "022404486c97c762a45a52dd9bc66e1d"
server_key_iv: "e6b0d07b2796067892e22d99710ac4fa"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:33.502948 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "bc98d3454e0549faafaf9caf085e06ce"
format_stamp: "Formatted at 2026-08-06 23:58:33 on dist-test-slave-39l8"
server_key: "022404486c97c762a45a52dd9bc66e1d"
server_key_iv: "e6b0d07b2796067892e22d99710ac4fa"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:33.507115 24616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:58:33.523646 24636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:33.524989 24616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:58:33.525128 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "bc98d3454e0549faafaf9caf085e06ce"
format_stamp: "Formatted at 2026-08-06 23:58:33 on dist-test-slave-39l8"
server_key: "022404486c97c762a45a52dd9bc66e1d"
server_key_iv: "e6b0d07b2796067892e22d99710ac4fa"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:33.525247 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:58:33.537490 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:58:33.540336 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:58:33.540555 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:58:33.548910 24616 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:41809
I20260806 23:58:33.548961 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:41809 every 8 connection(s)
I20260806 23:58:33.549908 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:58:33.552842 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:58:33.556313 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 24616
I20260806 23:58:33.556439 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:58:33.556756 9009 external_mini_cluster.cc:1428] Setting key 280e2e6246bded488e7078f7b1ec4437
I20260806 23:58:33.559305 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: Bootstrap starting.
I20260806 23:58:33.561944 24707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: Neither blocks nor log segments found. Creating new log.
I20260806 23:58:33.562883 24707 log.cc:824] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: Log is configured to *not* fsync() on all Append() calls
I20260806 23:58:33.565062 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: No bootstrap required, opened a new log
Aug 06 23:58:33 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:58:33.569285 24707 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } }
I20260806 23:58:33.569533 24707 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:58:33.569638 24707 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc98d3454e0549faafaf9caf085e06ce, State: Initialized, Role: FOLLOWER
I20260806 23:58:33.570165 24707 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } }
I20260806 23:58:33.570333 24707 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:58:33.570436 24707 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:58:33.570552 24707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:58:33.571662 24707 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } }
I20260806 23:58:33.572060 24707 leader_election.cc:302] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc98d3454e0549faafaf9caf085e06ce; no voters:
I20260806 23:58:33.572428 24707 leader_election.cc:288] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:58:33.572513 24714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:58:33.572829 24714 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [term 1 LEADER]: Becoming Leader. State: Replica: bc98d3454e0549faafaf9caf085e06ce, State: Running, Role: LEADER
I20260806 23:58:33.573673 24707 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:58:33.573781 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:33.558192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.8.204.126:41809)
0806 23:58:33.558646 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:33.558664 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:33.558729 (+ 65us) server_negotiation.cc:407] Connection header received
0806 23:58:33.559587 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:33.559610 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:33.559992 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:33.560331 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:58:33.561225 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:33.562799 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:33.564016 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:33.564698 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:33.564860 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:33.567950 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:58:33.567957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:58:33.567972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:58:33.568001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:58:33.569966 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:33.570485 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:33.570491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:33.570497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:33.570644 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:33.570998 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:33.571001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:33.571003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:33.571501 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:58:33.571707 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:33.572576 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 23:58:33.572854 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260806 23:58:33.573171 24714 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } }
I20260806 23:58:33.575991 24714 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc98d3454e0549faafaf9caf085e06ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } } }
I20260806 23:58:33.576124 24714 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [sys.catalog]: This master's current role is: LEADER
I20260806 23:58:33.576308 24714 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc98d3454e0549faafaf9caf085e06ce. Latest consensus state: current_term: 1 leader_uuid: "bc98d3454e0549faafaf9caf085e06ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98d3454e0549faafaf9caf085e06ce" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41809 } } }
I20260806 23:58:33.576331 24707 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:58:33.576419 24714 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce [sys.catalog]: This master's current role is: LEADER
I20260806 23:58:33.582340 24729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:58:33.586597 24729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:58:33.593056 24729 catalog_manager.cc:1347] Generated new cluster ID: a045e265ee4a4720ad28d84bbfad11c2
I20260806 23:58:33.593122 24729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:58:33.607563 24729 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:58:33.608860 24729 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:58:33.614792 24729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: Generated new TSK 0
I20260806 23:58:33.615466 24729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:58:34.309095 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:47061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:58:34.426112 24765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:58:34.426353 24765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:58:34.426415 24765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:58:34.430680 24765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:58:34.430792 24765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:58:34.430887 24765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:58:34.435989 24765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.204.84:47061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:58:34.436939 24765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:58:34.437842 24765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:58:34.446969 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:34.446978 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:34.447399 24777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:34.447636 24765 server_base.cc:1034] running on GCE node
I20260806 23:58:34.448009 24765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:58:34.448634 24765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:58:34.449806 24765 hybrid_clock.cc:648] HybridClock initialized: now 1786060714449788 us; error 36 us; skew 500 ppm
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:58:34.452706 24765 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:58:34.453619 24765 webserver.cc:460] Webserver started at http://127.8.204.65:43595/ using document root <none> and password file <none>
I20260806 23:58:34.454126 24765 fs_manager.cc:359] Metadata directory not provided
I20260806 23:58:34.454182 24765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:58:34.454376 24765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:58:34.469105 24765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "1ed2ea567172476380f143ef41a195c6"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "c1d01aab74bd891f24c654f28917819b"
server_key_iv: "a15157b137da5673a90581881e0c6465"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.469797 24765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "1ed2ea567172476380f143ef41a195c6"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "c1d01aab74bd891f24c654f28917819b"
server_key_iv: "a15157b137da5673a90581881e0c6465"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.476197 24765 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 23:58:34.498806 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.500537 24765 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260806 23:58:34.500675 24765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "1ed2ea567172476380f143ef41a195c6"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "c1d01aab74bd891f24c654f28917819b"
server_key_iv: "a15157b137da5673a90581881e0c6465"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.500806 24765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:58:34.536703 24765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:58:34.540575 24765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:58:34.540841 24765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:58:34.541611 24765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:58:34.542644 24765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:58:34.542737 24765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.542793 24765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:58:34.542858 24765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.557863 24765 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:34829
I20260806 23:58:34.559046 24765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:58:34.560278 24960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:34829 every 8 connection(s)
I20260806 23:58:34.567637 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 24765
I20260806 23:58:34.567761 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:58:34.568063 9009 external_mini_cluster.cc:1428] Setting key ebfa30815e97a3350eec7ed8a33dabb1
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:58:34.589494 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:34.562889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:51249 (local address 127.8.204.126:41809)
0806 23:58:34.564355 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:34.564370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:34.575177 (+ 10807us) server_negotiation.cc:407] Connection header received
0806 23:58:34.576398 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:34.576401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:34.576452 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:34.576514 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:58:34.579477 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.580335 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.581922 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.582275 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.582352 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:34.585155 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:58:34.585160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:58:34.585163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:58:34.585187 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:58:34.586992 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:34.587621 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:34.587625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:34.587626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:34.587721 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:34.588155 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:34.588158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:34.588160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:34.588415 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:58:34.588515 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:34.589174 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 23:58:34.589296 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":111,"threads_started":1}
I20260806 23:58:34.591140 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:34.563456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41809 (local address 127.8.204.65:51249)
0806 23:58:34.575000 (+ 11544us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:34.575059 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:34.576155 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:34.576907 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:34.576916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:34.577274 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:58:34.577895 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:34.577913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.580496 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.580499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:34.581772 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:34.581782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.582553 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.582556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:34.582786 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:34.583420 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:58:34.583439 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:58:34.584975 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:58:34.587144 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:34.587151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:34.587165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:34.587496 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:34.587838 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:34.587841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:34.587843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:34.588013 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:34.588588 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:58:34.588594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:58:34.588882 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 23:58:34.589831 (+ 949us) client_negotiation.cc:240] Negotiation successful
0806 23:58:34.590148 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11305,"thread_start_us":147,"threads_started":1}
I20260806 23:58:34.592715 24961 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41809
I20260806 23:58:34.592980 24961 heartbeater.cc:464] Registering TS with master...
I20260806 23:58:34.593580 24961 heartbeater.cc:511] Master 127.8.204.126:41809 requested a full tablet report, sending...
I20260806 23:58:34.595289 24657 ts_manager.cc:194] Registered new tserver with Master: 1ed2ea567172476380f143ef41a195c6 (127.8.204.65:34829)
I20260806 23:58:34.596825 24657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:51249
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:58:34.619613 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:47061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 23:58:34.756444 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:58:34.756762 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:58:34.756822 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:58:34.762307 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:58:34.762362 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:58:34.762415 24969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:58:34.768929 24969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.204.84:47061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:58:34.770141 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:58:34.771415 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:58:34.783229 24978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:34.783260 24980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:34.783730 24969 server_base.cc:1034] running on GCE node
W20260806 23:58:34.783671 24977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:34.784336 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:58:34.785061 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:58:34.786264 24969 hybrid_clock.cc:648] HybridClock initialized: now 1786060714786245 us; error 29 us; skew 500 ppm
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:58:34.789958 24969 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:58:34.790941 24969 webserver.cc:460] Webserver started at http://127.8.204.66:36161/ using document root <none> and password file <none>
I20260806 23:58:34.791591 24969 fs_manager.cc:359] Metadata directory not provided
I20260806 23:58:34.791700 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:58:34.791940 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:58:34.809425 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "8b0e95b1aafd497199bc6b3c1469b15b"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "6b34f488e164730f5a2df09d350735f1"
server_key_iv: "e075f99820c04b00d1823eca41985359"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.810226 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "8b0e95b1aafd497199bc6b3c1469b15b"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "6b34f488e164730f5a2df09d350735f1"
server_key_iv: "e075f99820c04b00d1823eca41985359"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.820128 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260806 23:58:34.841665 24989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.843302 24969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 23:58:34.843456 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "8b0e95b1aafd497199bc6b3c1469b15b"
format_stamp: "Formatted at 2026-08-06 23:58:34 on dist-test-slave-39l8"
server_key: "6b34f488e164730f5a2df09d350735f1"
server_key_iv: "e075f99820c04b00d1823eca41985359"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:34.843602 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:58:34.857033 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:58:34.860459 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:58:34.860667 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:58:34.861320 24969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:58:34.862329 24969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:58:34.862408 24969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.862460 24969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:58:34.862490 24969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:34.874243 24969 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:33523
I20260806 23:58:34.875655 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:58:34.876626 25132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:33523 every 8 connection(s)
I20260806 23:58:34.877995 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 24969
I20260806 23:58:34.878137 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:58:34.878507 9009 external_mini_cluster.cc:1428] Setting key 411edea2cb4e59257007dab71f2d1fdb
Aug 06 23:58:34 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:58:34.892874 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:34.880268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:44073 (local address 127.8.204.126:41809)
0806 23:58:34.880392 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:34.880396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:34.880412 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:58:34.881018 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:34.881024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:34.881085 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:34.881138 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:58:34.882638 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.883422 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.884997 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.885330 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.885460 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:34.888451 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:58:34.888454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:58:34.888457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:58:34.888483 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:58:34.890360 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:34.891053 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:34.891059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:34.891063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:34.891155 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:34.891571 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:34.891575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:34.891576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:34.891792 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:58:34.891879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:34.892559 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 23:58:34.892693 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:58:34.894515 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:34.879076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41809 (local address 127.8.204.66:44073)
0806 23:58:34.879659 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:34.879712 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:34.880758 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:34.881512 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:34.881521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:34.881843 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:58:34.882479 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:34.882497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.883576 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.883579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:34.884832 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:34.884842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:34.885678 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:34.885682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:34.885873 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:34.886541 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:58:34.886561 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:58:34.888254 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:58:34.890498 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:34.890504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:34.890517 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:34.890907 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:34.891276 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:34.891279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:34.891281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:34.891454 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:34.891954 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:58:34.891960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:58:34.892271 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 23:58:34.893103 (+ 832us) client_negotiation.cc:240] Negotiation successful
0806 23:58:34.893420 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":171,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:58:34.896914 25133 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41809
I20260806 23:58:34.897171 25133 heartbeater.cc:464] Registering TS with master...
I20260806 23:58:34.897708 25133 heartbeater.cc:511] Master 127.8.204.126:41809 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260806 23:58:34.898976 24657 ts_manager.cc:194] Registered new tserver with Master: 8b0e95b1aafd497199bc6b3c1469b15b (127.8.204.66:33523)
I20260806 23:58:34.899696 24657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:44073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:58:34.922785 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:47061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:58:35.085328 25138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:58:35.086087 25138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:58:35.086236 25138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:58:35.092144 25138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:58:35.092305 25138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:58:35.092445 25138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:58:35.099779 25138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.204.84:47061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:58:35.101212 25138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:58:35.102546 25138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:58:35.115963 25147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:58:35.116470 25146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:35.117324 25138 server_base.cc:1034] running on GCE node
W20260806 23:58:35.117653 25149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:58:35.118146 25138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:58:35.118935 25138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:58:35.120244 25138 hybrid_clock.cc:648] HybridClock initialized: now 1786060715120182 us; error 81 us; skew 500 ppm
Aug 06 23:58:35 dist-test-slave-39l8 krb5kdc[24026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:58:35.123970 25138 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:58:35.125030 25138 webserver.cc:460] Webserver started at http://127.8.204.67:44197/ using document root <none> and password file <none>
I20260806 23:58:35.125742 25138 fs_manager.cc:359] Metadata directory not provided
I20260806 23:58:35.125838 25138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:58:35.126076 25138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:58:35 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:58:35.145205 25138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "6d637f968e534005bfc05590ac55e97b"
format_stamp: "Formatted at 2026-08-06 23:58:35 on dist-test-slave-39l8"
server_key: "5f716a1eaaae53da241e0f5abb68aab5"
server_key_iv: "6d42c52ae46c7a27590051c3d05952ea"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:35.146289 25138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "6d637f968e534005bfc05590ac55e97b"
format_stamp: "Formatted at 2026-08-06 23:58:35 on dist-test-slave-39l8"
server_key: "5f716a1eaaae53da241e0f5abb68aab5"
server_key_iv: "6d42c52ae46c7a27590051c3d05952ea"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:35.152161 25138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 23:58:35.175791 25157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:35.177515 25138 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 23:58:35.177639 25138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "6d637f968e534005bfc05590ac55e97b"
format_stamp: "Formatted at 2026-08-06 23:58:35 on dist-test-slave-39l8"
server_key: "5f716a1eaaae53da241e0f5abb68aab5"
server_key_iv: "6d42c52ae46c7a27590051c3d05952ea"
server_key_version: "kuduclusterkey@0"
I20260806 23:58:35.177758 25138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:58:35.192091 25138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:58:35.195621 25138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:58:35.195856 25138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:58:35.196627 25138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:58:35.197736 25138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:58:35.197853 25138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:35.197948 25138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:58:35.198010 25138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:58:35.211876 25138 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:46785
I20260806 23:58:35.213083 25138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:58:35.214547 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:46785 every 8 connection(s)
I20260806 23:58:35.222635 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 25138
I20260806 23:58:35.222805 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:58:35.223124 9009 external_mini_cluster.cc:1428] Setting key 755b4034808479f00e3425709142809f
Aug 06 23:58:35 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:58:35 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:58:35.233094 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.218115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:40129 (local address 127.8.204.126:41809)
0806 23:58:35.218243 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.218248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.218861 (+ 613us) server_negotiation.cc:407] Connection header received
0806 23:58:35.220527 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.220531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.220592 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.220651 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:58:35.222259 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.223052 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.225352 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.225688 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.225770 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.228749 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:58:35.228753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:58:35.228756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:58:35.228784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:58:35.230618 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:35.231257 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:35.231260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:35.231262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:35.231351 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:35.231795 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:35.231798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:35.231800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:35.232041 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:58:35.232130 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.232799 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.232936 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:58:35.234731 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.216995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:41809 (local address 127.8.204.67:40129)
0806 23:58:35.218679 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.218755 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.220206 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.221056 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.221064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.221408 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:58:35.222072 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.222094 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.223830 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.223833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.225200 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.225212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.225974 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.225978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.226175 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.226822 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:58:35.226844 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:58:35.228562 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:58:35.230783 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:35.230789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:35.230802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:35.231131 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:35.231466 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:58:35.231469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:58:35.231471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:58:35.231660 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:58:35.232204 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:58:35.232210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:58:35.232517 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.233319 (+ 802us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.233675 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"thread_start_us":142,"threads_started":1}
I20260806 23:58:35.236364 25304 heartbeater.cc:347] Connected to a master server at 127.8.204.126:41809
I20260806 23:58:35.236632 25304 heartbeater.cc:464] Registering TS with master...
I20260806 23:58:35.237198 25304 heartbeater.cc:511] Master 127.8.204.126:41809 requested a full tablet report, sending...
I20260806 23:58:35.238355 24657 ts_manager.cc:194] Registered new tserver with Master: 6d637f968e534005bfc05590ac55e97b (127.8.204.67:46785)
I20260806 23:58:35.239182 24657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:40129
I20260806 23:58:35.249195 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:58:35.276335 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.255662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.8.204.126:41809)
0806 23:58:35.255792 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.255796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.264017 (+ 8221us) server_negotiation.cc:407] Connection header received
0806 23:58:35.264223 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.264227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.264278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.264329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:58:35.265444 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.266282 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.267474 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.267788 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.267864 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.269852 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:58:35.269856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:58:35.269859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:58:35.269886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:58:35.272623 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:35.273705 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:35.273711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:35.273716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:35.273812 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:58:35.275309 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:58:35.275325 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:58:35.275326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:58:35.275567 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:58:35.275674 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.276051 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.276168 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:58:35.280260 24657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:58:35.284896 24657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:58:35.356626 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.330126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:46785 (local address 127.0.0.1:39486)
0806 23:58:35.334676 (+ 4550us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.334692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.334829 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.335713 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.335719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.335742 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:58:35.335901 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.335907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.347454 (+ 11547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.347458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.348868 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.348882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.356019 (+ 7137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.356023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.356161 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.356209 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.356350 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.356475 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4479,"thread_start_us":99,"threads_started":1}
I20260806 23:58:35.357079 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.330549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.8.204.67:46785)
0806 23:58:35.335117 (+ 4568us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.335128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.335150 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:58:35.335223 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.335228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.335398 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.335541 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:58:35.337923 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.347300 (+ 9377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.350647 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.355855 (+ 5208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.355996 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.356200 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.356828 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.356932 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4449,"thread_start_us":102,"threads_started":1}
I20260806 23:58:35.359735 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.327302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:34829 (local address 127.0.0.1:53232)
0806 23:58:35.332179 (+ 4877us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.332235 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.332368 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.337665 (+ 5297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.337668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.337688 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:58:35.337780 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.337786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.340191 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.340195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.349695 (+ 9500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.349704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.359301 (+ 9597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.359305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.359410 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.359437 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.359526 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.359599 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4620,"thread_start_us":82,"threads_started":1}
I20260806 23:58:35.360958 25223 tablet_service.cc:1467] Processing CreateTablet for tablet 22a48dee693b4d148c0259920ac272c3 (DEFAULT_TABLE table=test-table [id=0108432b1681431f842876dc0d7ecea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:58:35.362010 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.329366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33523 (local address 127.0.0.1:44770)
0806 23:58:35.333460 (+ 4094us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.333474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.334445 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.351885 (+ 17440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.351888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.351911 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:58:35.352031 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.352040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.353703 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.353706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.354986 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.354999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.361550 (+ 6551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.361554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.361689 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.361731 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.361809 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.361878 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4010,"thread_start_us":91,"threads_started":1}
I20260806 23:58:35.362380 25223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a48dee693b4d148c0259920ac272c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:58:35.362893 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.328059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.8.204.65:34829)
0806 23:58:35.332663 (+ 4604us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.332671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.332697 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:58:35.332780 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.332785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.333004 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.333183 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:58:35.338806 (+ 5623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.340029 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.349837 (+ 9808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.356648 (+ 6811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.359042 (+ 2394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.359232 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.362574 (+ 3342us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.362735 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4488,"thread_start_us":91,"threads_started":1}
I20260806 23:58:35.368463 24832 tablet_service.cc:1467] Processing CreateTablet for tablet 22a48dee693b4d148c0259920ac272c3 (DEFAULT_TABLE table=test-table [id=0108432b1681431f842876dc0d7ecea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:58:35.369635 24832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a48dee693b4d148c0259920ac272c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:58:35.379185 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.329568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.8.204.66:33523)
0806 23:58:35.350970 (+ 21402us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.350993 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.351020 (+ 27us) server_negotiation.cc:407] Connection header received
0806 23:58:35.351105 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.351110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.351334 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.351513 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:58:35.352212 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.353534 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.360310 (+ 6776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.361214 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.361330 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.361483 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.378884 (+ 17401us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.379017 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21278,"thread_start_us":133,"threads_started":1}
I20260806 23:58:35.382301 25024 tablet_service.cc:1467] Processing CreateTablet for tablet 22a48dee693b4d148c0259920ac272c3 (DEFAULT_TABLE table=test-table [id=0108432b1681431f842876dc0d7ecea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:58:35.383351 25024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a48dee693b4d148c0259920ac272c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:58:35.391866 25334 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Bootstrap starting.
I20260806 23:58:35.394470 25334 tablet_bootstrap.cc:654] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Neither blocks nor log segments found. Creating new log.
I20260806 23:58:35.394930 25338 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Bootstrap starting.
I20260806 23:58:35.397204 25338 tablet_bootstrap.cc:654] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Neither blocks nor log segments found. Creating new log.
I20260806 23:58:35.398298 25334 log.cc:824] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Log is configured to *not* fsync() on all Append() calls
I20260806 23:58:35.399757 25338 log.cc:824] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:58:35.400880 25333 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Bootstrap starting.
I20260806 23:58:35.403712 25333 tablet_bootstrap.cc:654] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Neither blocks nor log segments found. Creating new log.
I20260806 23:58:35.404565 25333 log.cc:824] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:58:35.406610 25334 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: No bootstrap required, opened a new log
I20260806 23:58:35.406967 25334 ts_tablet_manager.cc:1397] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260806 23:58:35.412667 25334 raft_consensus.cc:348] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.414355 25338 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: No bootstrap required, opened a new log
I20260806 23:58:35.414589 25338 ts_tablet_manager.cc:1397] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260806 23:58:35.418645 25338 raft_consensus.cc:348] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.419185 25338 raft_consensus.cc:374] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:58:35.419302 25338 raft_consensus.cc:729] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0e95b1aafd497199bc6b3c1469b15b, State: Initialized, Role: FOLLOWER
I20260806 23:58:35.420178 25334 raft_consensus.cc:374] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:58:35.420351 25334 raft_consensus.cc:729] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed2ea567172476380f143ef41a195c6, State: Initialized, Role: FOLLOWER
I20260806 23:58:35.420845 25334 consensus_queue.cc:260] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.419902 25338 consensus_queue.cc:260] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.422048 25338 ts_tablet_manager.cc:1428] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:58:35.422997 25334 ts_tablet_manager.cc:1428] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260806 23:58:35.423494 25133 heartbeater.cc:503] Master 127.8.204.126:41809 was elected leader, sending a full tablet report...
I20260806 23:58:35.427733 24961 heartbeater.cc:503] Master 127.8.204.126:41809 was elected leader, sending a full tablet report...
I20260806 23:58:35.432686 25333 tablet_bootstrap.cc:492] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: No bootstrap required, opened a new log
I20260806 23:58:35.433001 25333 ts_tablet_manager.cc:1397] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20260806 23:58:35.437172 25333 raft_consensus.cc:348] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.439311 25333 raft_consensus.cc:374] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:58:35.439437 25333 raft_consensus.cc:729] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d637f968e534005bfc05590ac55e97b, State: Initialized, Role: FOLLOWER
I20260806 23:58:35.440147 25333 consensus_queue.cc:260] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.442775 25333 ts_tablet_manager.cc:1428] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20260806 23:58:35.443197 25304 heartbeater.cc:503] Master 127.8.204.126:41809 was elected leader, sending a full tablet report...
W20260806 23:58:35.468613 25305 tablet.cc:2369] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:58:35.503362 25346 raft_consensus.cc:482] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:58:35.503592 25346 raft_consensus.cc:504] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.505332 25346 leader_election.cc:288] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed2ea567172476380f143ef41a195c6 (127.8.204.65:34829), 8b0e95b1aafd497199bc6b3c1469b15b (127.8.204.66:33523)
I20260806 23:58:35.510159 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.505205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:34829 (local address 127.8.204.67:48441)
0806 23:58:35.505505 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.505520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.505616 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.506233 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.506236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.506255 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:58:35.506368 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.506374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.507390 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.507393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.508687 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.508696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.509723 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.509726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.509839 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.509862 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.509943 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.510025 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":31}
I20260806 23:58:35.510644 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.505629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:48441 (local address 127.8.204.65:34829)
0806 23:58:35.505779 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.505783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.505797 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:58:35.505849 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.505851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.505899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.505984 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:58:35.506496 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.507237 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.508869 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.509466 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.509587 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.509654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.510438 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.510523 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 23:58:35.511288 24873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a48dee693b4d148c0259920ac272c3" candidate_uuid: "6d637f968e534005bfc05590ac55e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed2ea567172476380f143ef41a195c6" is_pre_election: true
I20260806 23:58:35.511618 24873 raft_consensus.cc:2454] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d637f968e534005bfc05590ac55e97b in term 0.
I20260806 23:58:35.511716 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.505770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:44399 (local address 127.8.204.66:33523)
0806 23:58:35.506178 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.506182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.506932 (+ 750us) server_negotiation.cc:407] Connection header received
0806 23:58:35.507095 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.507098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.507141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.507196 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:58:35.507690 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.508392 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.509848 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.510440 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.510539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.510584 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.511534 (+ 950us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.511601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260806 23:58:35.512167 25161 leader_election.cc:302] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed2ea567172476380f143ef41a195c6, 6d637f968e534005bfc05590ac55e97b; no voters:
I20260806 23:58:35.512437 25346 raft_consensus.cc:2781] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:58:35.512524 25346 raft_consensus.cc:482] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:58:35.512569 25346 raft_consensus.cc:3037] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:58:35.512733 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.505205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33523 (local address 127.8.204.67:44399)
0806 23:58:35.506855 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.506867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.506992 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.507370 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:58:35.507372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:58:35.507390 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:58:35.507559 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.507565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.508544 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.508547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.509729 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:58:35.509737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.511278 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.511283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:58:35.511372 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.511399 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:58:35.512440 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0806 23:58:35.512527 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"mutex_wait_us":644,"thread_start_us":102,"threads_started":1}
I20260806 23:58:35.513372 25055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a48dee693b4d148c0259920ac272c3" candidate_uuid: "6d637f968e534005bfc05590ac55e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" is_pre_election: true
I20260806 23:58:35.513643 25055 raft_consensus.cc:2454] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d637f968e534005bfc05590ac55e97b in term 0.
I20260806 23:58:35.514056 25346 raft_consensus.cc:504] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.514374 25346 leader_election.cc:288] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [CANDIDATE]: Term 1 election: Requested vote from peers 1ed2ea567172476380f143ef41a195c6 (127.8.204.65:34829), 8b0e95b1aafd497199bc6b3c1469b15b (127.8.204.66:33523)
I20260806 23:58:35.515158 25055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a48dee693b4d148c0259920ac272c3" candidate_uuid: "6d637f968e534005bfc05590ac55e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0e95b1aafd497199bc6b3c1469b15b"
I20260806 23:58:35.515297 25055 raft_consensus.cc:3037] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:58:35.516247 25055 raft_consensus.cc:2454] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d637f968e534005bfc05590ac55e97b in term 1.
I20260806 23:58:35.516613 25160 leader_election.cc:302] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d637f968e534005bfc05590ac55e97b, 8b0e95b1aafd497199bc6b3c1469b15b; no voters:
I20260806 23:58:35.516628 24873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a48dee693b4d148c0259920ac272c3" candidate_uuid: "6d637f968e534005bfc05590ac55e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed2ea567172476380f143ef41a195c6"
I20260806 23:58:35.516736 24873 raft_consensus.cc:3037] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:58:35.516850 25346 raft_consensus.cc:2781] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:58:35.517071 25346 raft_consensus.cc:686] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 1 LEADER]: Becoming Leader. State: Replica: 6d637f968e534005bfc05590ac55e97b, State: Running, Role: LEADER
I20260806 23:58:35.517391 25346 consensus_queue.cc:237] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } }
I20260806 23:58:35.518054 24873 raft_consensus.cc:2454] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d637f968e534005bfc05590ac55e97b in term 1.
I20260806 23:58:35.520825 24657 catalog_manager.cc:5495] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d637f968e534005bfc05590ac55e97b (127.8.204.67). New cstate: current_term: 1 leader_uuid: "6d637f968e534005bfc05590ac55e97b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d637f968e534005bfc05590ac55e97b" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46785 } health_report { overall_health: HEALTHY } } }
Aug 06 23:58:35 dist-test-slave-39l8 krb5kdc[24026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
W20260806 23:58:35.565398 24962 tablet.cc:2369] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:58:35.580051 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.575062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.8.204.67:46785)
0806 23:58:35.575246 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 23:58:35.575251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:58:35.575266 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:58:35.575355 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:58:35.575361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:58:35.575443 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:58:35.575564 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:58:35.575873 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.576554 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.577722 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:58:35.578028 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:58:35.578147 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:58:35.578489 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:58:35.579054 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:58:35.579203 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:58:35.579414 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 23:58:35.579881 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 23:58:35.587433 24873 raft_consensus.cc:1261] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 1 FOLLOWER]: Refusing update from remote peer 6d637f968e534005bfc05590ac55e97b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:58:35.587472 25055 raft_consensus.cc:1261] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 1 FOLLOWER]: Refusing update from remote peer 6d637f968e534005bfc05590ac55e97b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:58:35.588317 25350 consensus_queue.cc:1035] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0e95b1aafd497199bc6b3c1469b15b" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:58:35.588526 25346 consensus_queue.cc:1035] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed2ea567172476380f143ef41a195c6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:58:35.601500 25357 mvcc.cc:205] Tried to move back new op lower bound from 7315704691037921280 to 7315704690762280960. Current Snapshot: MvccSnapshot[applied={T|T < 7315704691037921280}]
I20260806 23:58:35.604421 25356 mvcc.cc:205] Tried to move back new op lower bound from 7315704691037921280 to 7315704690762280960. Current Snapshot: MvccSnapshot[applied={T|T < 7315704691037921280 or (T in {7315704691037921280})}]
I20260806 23:58:35.607987 24656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:58:35.608134 24656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:58:35.610540 24656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc98d3454e0549faafaf9caf085e06ce: Sending DeleteTablet for 3 replicas of tablet 22a48dee693b4d148c0259920ac272c3
I20260806 23:58:35.611296 24832 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a48dee693b4d148c0259920ac272c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:58:35 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53232
I20260806 23:58:35.611745 25366 tablet_replica.cc:330] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6: stopping tablet replica
I20260806 23:58:35.611878 25223 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a48dee693b4d148c0259920ac272c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:58:35 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:39486
I20260806 23:58:35.612146 25366 raft_consensus.cc:2229] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:58:35.612439 25366 raft_consensus.cc:2258] T 22a48dee693b4d148c0259920ac272c3 P 1ed2ea567172476380f143ef41a195c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:58:35.612542 25024 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a48dee693b4d148c0259920ac272c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:58:35 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44770
I20260806 23:58:35.613023 25369 tablet_replica.cc:330] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: stopping tablet replica
I20260806 23:58:35.613329 25369 raft_consensus.cc:2229] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:58:35.613620 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 24765
I20260806 23:58:35.613639 25369 raft_consensus.cc:2258] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:58:35.624599 25369 ts_tablet_manager.cc:1905] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:58:35.628794 25369 ts_tablet_manager.cc:1918] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:58:35.629096 25368 tablet_replica.cc:330] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: stopping tablet replica
I20260806 23:58:35.629436 25368 raft_consensus.cc:2229] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:58:35.629839 25368 raft_consensus.cc:2258] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 23:58:35.629954 24641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:34829 (error 108)
I20260806 23:58:35.630134 25369 log.cc:1196] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/22a48dee693b4d148c0259920ac272c3
I20260806 23:58:35.630463 25369 ts_tablet_manager.cc:1939] T 22a48dee693b4d148c0259920ac272c3 P 8b0e95b1aafd497199bc6b3c1469b15b: Deleting consensus metadata
I20260806 23:58:35.630966 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 24969
I20260806 23:58:35.643822 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.643457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:34829 (local address 127.0.0.1:53242)
0806 23:58:35.643651 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.643737 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:34829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260806 23:58:35.644004 24641 catalog_manager.cc:4584] TS 1ed2ea567172476380f143ef41a195c6 (127.8.204.65:34829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22a48dee693b4d148c0259920ac272c3: Network error: Client connection negotiation failed: client connection to 127.8.204.65:34829: connect: Connection refused (error 111)
I20260806 23:58:35.644265 25368 ts_tablet_manager.cc:1905] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:58:35.646916 25368 ts_tablet_manager.cc:1918] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:58:35.647066 25368 log.cc:1196] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/22a48dee693b4d148c0259920ac272c3
I20260806 23:58:35.647528 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:58:35.647234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:33523 (local address 127.0.0.1:44772)
0806 23:58:35.647398 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.647455 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.66:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 23:58:35.647600 25368 ts_tablet_manager.cc:1939] T 22a48dee693b4d148c0259920ac272c3 P 6d637f968e534005bfc05590ac55e97b: Deleting consensus metadata
I20260806 23:58:35.647737 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 25138
W20260806 23:58:35.656339 25322 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:58:35.655789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:46785 (local address 127.0.0.1:39498)
0806 23:58:35.655935 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 23:58:35.655949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:58:35.656040 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:58:35.656259 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.67:46785: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 23:58:35.656831 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 24616
I20260806 23:58:35.670262 9009 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 23:58:36.037220 9009 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 23:58:36.037386 9009 mini_ranger.cc:66] Stopping Ranger...
I20260806 23:58:36.383507 9009 mini_ranger.cc:68] Stopped Ranger
2026-08-06 23:58:36.384 UTC [24061] LOG: received smart shutdown request
2026-08-06 23:58:36.387 UTC [24061] LOG: background worker "logical replication launcher" (PID 24074) exited with exit code 1
2026-08-06 23:58:36.388 UTC [24067] LOG: shutting down
2026-08-06 23:58:36.528 UTC [24061] LOG: database system is shut down
2026-08-06T23:58:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44716 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:58:36 dist-test-slave-39l8 krb5kdc[25387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:58:36 dist-test-slave-39l8 krb5kdc[25387](info): set up 2 sockets
Aug 06 23:58:36 dist-test-slave-39l8 krb5kdc[25387](info): commencing operation
krb5kdc: starting...
W20260806 23:58:37.797488 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:58:37 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:58:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangeradmin_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangeradmin_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangerlookup_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangerlookup_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
I20260806 23:58:37.982656 9009 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcJxSMk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/postgres -l logfile start
2026-08-06 23:58:40.288 UTC [25427] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 23:58:40.288 UTC [25427] LOG: listening on IPv4 address "127.8.204.84", port 54269
2026-08-06 23:58:40.291 UTC [25427] LOG: listening on Unix socket "/tmp/.s.PGSQL.54269"
2026-08-06 23:58:40.311 UTC [25430] LOG: database system was shut down at 2026-08-06 23:58:39 UTC
2026-08-06 23:58:40.315 UTC [25427] LOG: database system is ready to accept connections
I20260806 23:58:41.422824 9009 mini_postgres.cc:96] Postgres bound to 54269
2026-08-06 23:58:41.430 UTC [25443] FATAL: database "slave" does not exist
127.8.204.84:54269 - accepting connections
I20260806 23:58:41.431197 9009 mini_ranger.cc:161] Starting Ranger...
I20260806 23:58:41.447574 9009 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 23:58:41.847626 9009 mini_ranger.cc:87] Created ranger Postgres database
I20260806 23:58:41.847716 9009 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin
2026-08-06 23:58:42,114 [I] DB FLAVOR :POSTGRES
2026-08-06 23:58:42,114 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:58:42,114 [I] Checking connection..
2026-08-06 23:58:42,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 23:58:42,360 [I] Checking connection passed.
2026-08-06 23:58:42,361 [I] --------- Verifying version history table ---------
2026-08-06 23:58:42,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:58:42,630 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:58:42,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:58:42,899 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:58:42,899 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 23:58:42,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 23:58:43.126 UTC [25529] WARNING: there is no transaction in progress
2026-08-06 23:58:43,134 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 23:58:43,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:58:43,403 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 23:58:43,403 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:58:43,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 23:58:43,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 23:58:43,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-39l8.c.gcp-upstream.internal','N') ;"
2026-08-06 23:58:44,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:58:44,435 [I] Table x_portal_user does not exist in database ranger
2026-08-06 23:58:44,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:58:44,674 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 23:58:44,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:58:44,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:58:45,189 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 23:58:45,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 23:58:46.127 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.142 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.156 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.170 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.184 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.200 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.255 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.261 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.274 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.283 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.292 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.305 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.314 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.319 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.325 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46.331 UTC [25674] WARNING: there is no transaction in progress
2026-08-06 23:58:46,888 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 23:58:46,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 23:58:47,129 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 23:58:47,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:58:47,370 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 23:58:47,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:58:47,641 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 23:58:47,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:58:47,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:58:48,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-39l8.c.gcp-upstream.internal';"
2026-08-06 23:58:48,426 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 23:58:48,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 23:58:48,683 [I] DB_PATCHES have already been applied
I20260806 23:58:48.689625 9009 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcJxSMk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 23:58:48.689688 9009 mini_ranger.cc:193] Using host: 127.8.204.84
I20260806 23:58:48.694137 9009 mini_ranger.cc:227] Ranger admin URL: http://127.8.204.84:41509
Aug 06, 2026 11:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:58:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:58:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:58:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:58:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:58:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 23:58:50.025408 9009 mini_ranger.cc:160] Time spent starting Ranger: real 8.594s user 0.000s sys 0.012s
Aug 06, 2026 11:59:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:59:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 11:59:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:59:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:59:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 23:59:11.382932 9009 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangerkms_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/rangerkms_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/HTTP_127.8.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 23:59:11 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060751, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 23:59:11.474014 9009 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 23:59:11.484833 9009 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 23:59:12.302780 9009 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 23:59:12.302875 9009 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms
I20260806 23:59:12.302894 9009 mini_ranger_kms.cc:227] Using postgres at 127.8.204.84:54269
2026-08-06 23:59:12,485 [I] DB FLAVOR :POSTGRES
2026-08-06 23:59:12,485 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:59:12,486 [I] Checking connection
2026-08-06 23:59:12,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 23:59:12,766 [I] connection success
2026-08-06 23:59:12,767 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 23:59:12,767 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 23:59:12,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 23:59:13,040 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 23:59:13,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:59:13,041 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 23:59:13,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.84:54269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 23:59:13,317 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 23:59:13.833650 9009 mini_ranger_kms.cc:310] Created kms service
I20260806 23:59:14.015833 9009 mini_ranger_kms.cc:326] Created kudu user
I20260806 23:59:14.064069 9009 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 23:59:14 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060751, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:14.501976 9009 mini_ranger_kms.cc:384] Added ranger policy
I20260806 23:59:14.502131 9009 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcJxSMk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcJxSMk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 23:59:14.502157 9009 mini_ranger_kms.cc:240] Using host: 127.8.204.84
I20260806 23:59:14.507696 9009 mini_ranger_kms.cc:276] Ranger KMS PID: 25931
I20260806 23:59:14.507791 9009 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.204.84:55441
Aug 06, 2026 11:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:59:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:59:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 23:59:17 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:17.163815 9009 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.690s user 0.003s sys 0.164s
I20260806 23:59:17.163964 9009 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 23:59:17.164069 9009 mini_ranger_kms.cc:401] 127.8.204.84:55441/kms/v1/keys
Aug 06 23:59:17 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:59:17 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060757, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.84@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:59:20 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:59:20.153055 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36339
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:36339
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:20.272955 26003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:20.273253 26003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:20.273294 26003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:20.277515 26003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:20.277585 26003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:20.277606 26003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:20.277622 26003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:20.277638 26003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:20.283593 26003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.204.84:55441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:36339
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:20.284773 26003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:20.285885 26003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:20.296259 26010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:20.296401 26011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:20.296271 26016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:20.296695 26003 server_base.cc:1034] running on GCE node
I20260806 23:59:20.297369 26003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:20.298657 26003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:20.299907 26003 hybrid_clock.cc:648] HybridClock initialized: now 1786060760299867 us; error 57 us; skew 500 ppm
Aug 06 23:59:20 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060760, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:20.303103 26003 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:20.304206 26003 webserver.cc:460] Webserver started at http://127.8.204.126:37639/ using document root <none> and password file <none>
I20260806 23:59:20.304803 26003 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:20.304890 26003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:20.305123 26003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:59:20 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060760, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:20.368112 26003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "719999eca869445d9815e39b9cfb7fbd"
format_stamp: "Formatted at 2026-08-06 23:59:20 on dist-test-slave-39l8"
server_key: "aa93821fe27ed85fd91e7c614613f002"
server_key_iv: "e50c0fe774dcf01646b2dd563163e8fd"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:20.368789 26003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "719999eca869445d9815e39b9cfb7fbd"
format_stamp: "Formatted at 2026-08-06 23:59:20 on dist-test-slave-39l8"
server_key: "aa93821fe27ed85fd91e7c614613f002"
server_key_iv: "e50c0fe774dcf01646b2dd563163e8fd"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:20.373247 26003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:59:20.388016 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:20.389508 26003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:59:20.389688 26003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "719999eca869445d9815e39b9cfb7fbd"
format_stamp: "Formatted at 2026-08-06 23:59:20 on dist-test-slave-39l8"
server_key: "aa93821fe27ed85fd91e7c614613f002"
server_key_iv: "e50c0fe774dcf01646b2dd563163e8fd"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:20.389859 26003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:20.421212 26003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:20.424467 26003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:20.424707 26003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:20.433833 26003 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:36339
I20260806 23:59:20.433876 26083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:36339 every 8 connection(s)
I20260806 23:59:20.435099 26003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:20.438087 26085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:20.443786 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26003
I20260806 23:59:20.443928 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:59:20.444226 9009 external_mini_cluster.cc:1428] Setting key 80b9a835c854f275f334564b6c39da28
I20260806 23:59:20.444310 26085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Bootstrap starting.
I20260806 23:59:20.447464 26085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:20.448432 26085 log.cc:824] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:20.450874 26085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: No bootstrap required, opened a new log
Aug 06 23:59:20 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:20.455618 26085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:20.455904 26085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:20.455993 26085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719999eca869445d9815e39b9cfb7fbd, State: Initialized, Role: FOLLOWER
I20260806 23:59:20.456521 26085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:20.456678 26085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:20.456763 26085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:20.456866 26085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:20.458002 26085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:20.458389 26085 leader_election.cc:302] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 719999eca869445d9815e39b9cfb7fbd; no voters:
I20260806 23:59:20.458812 26085 leader_election.cc:288] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:59:20.458897 26090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:20.459208 26090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 LEADER]: Becoming Leader. State: Replica: 719999eca869445d9815e39b9cfb7fbd, State: Running, Role: LEADER
I20260806 23:59:20.459563 26090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:20.460016 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:20.445776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.8.204.126:36339)
0806 23:59:20.446297 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:20.446317 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:20.446362 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:59:20.447231 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:20.447259 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:20.447730 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:20.448114 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:20.449029 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:20.450082 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:20.451084 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:20.451552 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:20.451761 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:20.454435 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:20.454441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:20.454455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:20.454484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:20.456427 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:20.456986 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:20.456997 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:20.457001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:20.457109 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:20.457431 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:20.457434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:20.457436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:20.457827 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:20.458039 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:20.458684 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 23:59:20.459038 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":153,"threads_started":1}
I20260806 23:59:20.460196 26085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:20.461022 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "719999eca869445d9815e39b9cfb7fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } } }
I20260806 23:59:20.462539 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:20.462859 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 719999eca869445d9815e39b9cfb7fbd. Latest consensus state: current_term: 1 leader_uuid: "719999eca869445d9815e39b9cfb7fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } } }
I20260806 23:59:20.462986 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:20.463424 26085 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:59:20.468096 26106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:20.472579 26106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:20.480706 26106 catalog_manager.cc:1347] Generated new cluster ID: e0568150739a42d8b918dcece05df0ee
I20260806 23:59:20.480777 26106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:20.495924 26106 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:59:20.497411 26106 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:20.502969 26106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Generated new TSK 0
I20260806 23:59:20.503933 26106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:59:21.204155 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:21.352319 26156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:21.352771 26156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:21.352895 26156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:21.359331 26156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:21.359421 26156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:21.359512 26156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:21.367902 26156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:21.369313 26156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:21.370755 26156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:21.383459 26164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:21.383790 26165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:21.383481 26171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:21.384131 26156 server_base.cc:1034] running on GCE node
I20260806 23:59:21.384632 26156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:21.385462 26156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:21.386794 26156 hybrid_clock.cc:648] HybridClock initialized: now 1786060761386775 us; error 31 us; skew 500 ppm
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:21.390291 26156 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:21.391245 26156 webserver.cc:460] Webserver started at http://127.8.204.65:36553/ using document root <none> and password file <none>
I20260806 23:59:21.391829 26156 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:21.391923 26156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:21.392208 26156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:21.412901 26156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "d6a20c028f9648e2a6045c7269d14da6"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "5b3c6ba20e2cc0bd4ba280978b59ff97"
server_key_iv: "69566ff5614329bd80eb096d8dc8526d"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.413812 26156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a20c028f9648e2a6045c7269d14da6"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "5b3c6ba20e2cc0bd4ba280978b59ff97"
server_key_iv: "69566ff5614329bd80eb096d8dc8526d"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.424880 26156 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260806 23:59:21.456496 26183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.458300 26156 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260806 23:59:21.458495 26156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "d6a20c028f9648e2a6045c7269d14da6"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "5b3c6ba20e2cc0bd4ba280978b59ff97"
server_key_iv: "69566ff5614329bd80eb096d8dc8526d"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.458648 26156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:21.499933 26156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:21.503753 26156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:21.503991 26156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:21.504666 26156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:21.505658 26156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:21.505753 26156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.505847 26156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:21.505887 26156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.517472 26156 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:36271
I20260806 23:59:21.517515 26352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:36271 every 8 connection(s)
I20260806 23:59:21.518487 26156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:59:21.521747 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26156
I20260806 23:59:21.521860 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:59:21.522142 9009 external_mini_cluster.cc:1428] Setting key 711641882406ea976188aabda173d5bd
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:59:21.538416 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:21.520916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:39639 (local address 127.8.204.126:36339)
0806 23:59:21.524387 (+ 3471us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:21.524393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:21.524410 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:59:21.524455 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:21.524460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:21.524516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:21.524606 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:21.526343 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.527277 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.528651 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.529048 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.529168 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:21.532612 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:21.532616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:21.532619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:21.532656 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:21.534824 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:21.535603 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:21.535607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:21.535609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:21.535720 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:21.536211 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:21.536214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:21.536216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:21.536482 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:21.536591 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:21.538111 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0806 23:59:21.538250 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3364,"thread_start_us":93,"threads_started":1}
I20260806 23:59:21.538892 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:21.521341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.65:39639)
0806 23:59:21.522857 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:21.522916 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:21.523940 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:21.524957 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:21.524969 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:21.525349 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:21.526140 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:21.526165 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.527439 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.527445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:21.528509 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:21.528520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.529386 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.529393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:21.529628 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:21.530376 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:21.530403 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:21.532404 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:21.534987 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:21.534995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:21.535021 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:21.535463 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:21.535843 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:21.535850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:21.535852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:21.536072 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:21.536660 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:21.536667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:21.537077 (+ 410us) client_negotiation.cc:769] Sending connection context
0806 23:59:21.537326 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 23:59:21.537718 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":143,"threads_started":1}
I20260806 23:59:21.540742 26353 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260806 23:59:21.541011 26353 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:21.541649 26353 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:21.543620 26045 ts_manager.cc:194] Registered new tserver with Master: d6a20c028f9648e2a6045c7269d14da6 (127.8.204.65:36271)
I20260806 23:59:21.545222 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:39639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:59:21.558956 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 23:59:21.695534 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:21.695832 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:21.695951 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:21.700488 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:21.700690 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:21.700809 26360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:21.707608 26360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:21.709025 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:21.710261 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:21.723470 26367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:21.723848 26370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:21.724228 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:21.724416 26360 server_base.cc:1034] running on GCE node
I20260806 23:59:21.725039 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:21.725881 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:21.729209 26360 hybrid_clock.cc:648] HybridClock initialized: now 1786060761729126 us; error 91 us; skew 500 ppm
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:21.733230 26360 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:21.734628 26360 webserver.cc:460] Webserver started at http://127.8.204.66:43807/ using document root <none> and password file <none>
I20260806 23:59:21.735462 26360 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:21.735620 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:21.735906 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:21.759974 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "803b60ffcf294917ab6eff8bfd3d1d8e"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "a229137f73703e47876da245e7785198"
server_key_iv: "a51513f43bfee20f25210cb2b7371e37"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.760793 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "803b60ffcf294917ab6eff8bfd3d1d8e"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "a229137f73703e47876da245e7785198"
server_key_iv: "a51513f43bfee20f25210cb2b7371e37"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.767211 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 23:59:21.793485 26381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.795498 26360 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260806 23:59:21.795670 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "803b60ffcf294917ab6eff8bfd3d1d8e"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "a229137f73703e47876da245e7785198"
server_key_iv: "a51513f43bfee20f25210cb2b7371e37"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:21.795830 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:21.820990 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:21.824687 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:21.824899 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:21.825567 26360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:21.826632 26360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:21.826697 26360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.826795 26360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:21.826819 26360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:21.837814 26360 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:36761
I20260806 23:59:21.838985 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:59:21.840909 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26360
I20260806 23:59:21.841037 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:59:21.841359 9009 external_mini_cluster.cc:1428] Setting key 88033955595a146dad47886fcd527bb2
I20260806 23:59:21.849207 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:36761 every 8 connection(s)
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Aug 06 23:59:21 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:21.877264 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:21.856792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:52957 (local address 127.8.204.126:36339)
0806 23:59:21.856944 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:21.856949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:21.863292 (+ 6343us) server_negotiation.cc:407] Connection header received
0806 23:59:21.864558 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:21.864563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:21.864622 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:21.864721 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:21.866226 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.867059 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.868888 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.869235 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.869317 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:21.872712 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:21.872716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:21.872719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:21.872749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:21.874649 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:21.875303 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:21.875307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:21.875309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:21.875401 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:21.875833 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:21.875836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:21.875838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:21.876067 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:21.876186 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:21.876877 (+ 691us) server_negotiation.cc:299] Negotiation successful
0806 23:59:21.877044 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:59:21.878213 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:21.857358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.66:52957)
0806 23:59:21.863082 (+ 5724us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:21.863146 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:21.864272 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:21.864939 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:21.864948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:21.865321 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:21.866046 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:21.866067 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.867357 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.867361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:21.868739 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:21.868752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:21.869564 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:21.869568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:21.869772 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:21.870573 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:21.870594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:21.872509 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:21.874801 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:21.874807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:21.874824 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:21.875184 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:21.875520 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:21.875524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:21.875526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:21.875719 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:21.876202 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:21.876209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:21.876562 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 23:59:21.876798 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 23:59:21.877146 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5455,"thread_start_us":169,"threads_started":1}
I20260806 23:59:21.879678 26546 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
I20260806 23:59:21.879938 26546 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:21.880551 26546 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:21.881807 26045 ts_manager.cc:194] Registered new tserver with Master: 803b60ffcf294917ab6eff8bfd3d1d8e (127.8.204.66:36761)
I20260806 23:59:21.882546 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:52957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:59:21.891635 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:22.065796 26553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:22.066157 26553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:22.066231 26553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:22.072670 26553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:22.072743 26553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:22.072839 26553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:22.080430 26553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:22.081861 26553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:22.083416 26553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:22.099954 26566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.110081 26563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.110307 26564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:22.111572 26553 server_base.cc:1034] running on GCE node
I20260806 23:59:22.112387 26553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:22.113312 26553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:22.120712 26553 hybrid_clock.cc:648] HybridClock initialized: now 1786060762120617 us; error 97 us; skew 500 ppm
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:22.124769 26553 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:22.125958 26553 webserver.cc:460] Webserver started at http://127.8.204.67:41309/ using document root <none> and password file <none>
I20260806 23:59:22.126753 26553 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:22.126850 26553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:22.127102 26553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:22.154750 26553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "0ea9574928594e8ba9eb11de08bb114a"
format_stamp: "Formatted at 2026-08-06 23:59:22 on dist-test-slave-39l8"
server_key: "86776b20f4bab5c54855f513a454ad32"
server_key_iv: "7f813cc10bc86da69025942735337209"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:22.155804 26553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "0ea9574928594e8ba9eb11de08bb114a"
format_stamp: "Formatted at 2026-08-06 23:59:22 on dist-test-slave-39l8"
server_key: "86776b20f4bab5c54855f513a454ad32"
server_key_iv: "7f813cc10bc86da69025942735337209"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:22.162984 26553 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 23:59:22.186829 26575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.188565 26553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:59:22.188710 26553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "0ea9574928594e8ba9eb11de08bb114a"
format_stamp: "Formatted at 2026-08-06 23:59:22 on dist-test-slave-39l8"
server_key: "86776b20f4bab5c54855f513a454ad32"
server_key_iv: "7f813cc10bc86da69025942735337209"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:22.188860 26553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:22.274353 26553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:22.277976 26553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:22.278234 26553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:22.278991 26553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:22.280078 26553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:22.280174 26553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.280256 26553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:22.280303 26553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.290593 26553 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:38125
I20260806 23:59:22.291916 26553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:59:22.294936 26745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:38125 every 8 connection(s)
I20260806 23:59:22.299618 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26553
I20260806 23:59:22.299757 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:59:22.300101 9009 external_mini_cluster.cc:1428] Setting key ac5d410ade909fef627fdf398e7e8718
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:22.308949 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:22.296487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:43905 (local address 127.8.204.126:36339)
0806 23:59:22.296606 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:22.296611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:22.296629 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:59:22.297674 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:22.297678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:22.297741 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:22.297850 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:22.299529 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.300436 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.302013 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.302279 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.302406 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:22.305152 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:22.305157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:22.305159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:22.305191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:22.306803 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:22.307333 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:22.307336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:22.307338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:22.307424 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:22.307790 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:22.307793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:22.307795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:22.307999 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:22.308112 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:22.308638 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 23:59:22.308771 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:59:22.309763 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:22.295138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.67:43905)
0806 23:59:22.296082 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:22.296150 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:22.297391 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:22.298084 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:22.298095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:22.298509 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:22.299302 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:22.299324 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.300575 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.300578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:22.301876 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:22.301887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.302469 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.302473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:22.302692 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:22.303428 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:22.303445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:22.304978 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:22.306929 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:22.306935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:22.306952 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:22.307218 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:22.307526 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:22.307529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:22.307531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:22.307682 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:22.308113 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:22.308122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:22.308375 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 23:59:22.308575 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 23:59:22.308900 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":139,"threads_started":1}
I20260806 23:59:22.310999 26746 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
I20260806 23:59:22.311281 26746 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:22.311924 26746 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:22.313297 26045 ts_manager.cc:194] Registered new tserver with Master: 0ea9574928594e8ba9eb11de08bb114a (127.8.204.67:38125)
I20260806 23:59:22.314199 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:43905
I20260806 23:59:22.314684 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:59:22.315546 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26156
I20260806 23:59:22.338335 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26360
I20260806 23:59:22.354532 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26553
I20260806 23:59:22.365002 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26003
I20260806 23:59:22.378525 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36339
--webserver_interface=127.8.204.126
--webserver_port=37639
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:36339
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:22.491604 26751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:22.491864 26751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:22.491904 26751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:22.495870 26751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:22.495925 26751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:22.495945 26751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:22.495961 26751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:22.495976 26751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:22.501283 26751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:36339
--ranger_config_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=37639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:22.502192 26751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:22.503150 26751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:22.511543 26762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.511555 26765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.511554 26763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:22.512001 26751 server_base.cc:1034] running on GCE node
I20260806 23:59:22.512491 26751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:22.513468 26751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:22.514654 26751 hybrid_clock.cc:648] HybridClock initialized: now 1786060762514620 us; error 33 us; skew 500 ppm
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:22.517468 26751 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:22.518354 26751 webserver.cc:460] Webserver started at http://127.8.204.126:37639/ using document root <none> and password file <none>
I20260806 23:59:22.518970 26751 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:22.519066 26751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:22.568188 26751 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:59:22.570622 26777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.572026 26751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:59:22.572206 26751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "719999eca869445d9815e39b9cfb7fbd"
format_stamp: "Formatted at 2026-08-06 23:59:20 on dist-test-slave-39l8"
server_key: "aa93821fe27ed85fd91e7c614613f002"
server_key_iv: "e50c0fe774dcf01646b2dd563163e8fd"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:22.572788 26751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:22.592368 26751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:22.595111 26751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:22.595328 26751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:22.602818 26751 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:36339
I20260806 23:59:22.602833 26852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:36339 every 8 connection(s)
I20260806 23:59:22.603935 26751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:22.606343 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26751
I20260806 23:59:22.606841 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:36271
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:36339
--webserver_port=36553
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:22.608484 26856 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:59:22.610164 26856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Bootstrap starting.
I20260806 23:59:22.627336 26856 log.cc:824] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:22.633836 26856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:59:22.634253 26856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Bootstrap complete.
I20260806 23:59:22.638499 26856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:22.638970 26856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719999eca869445d9815e39b9cfb7fbd, State: Initialized, Role: FOLLOWER
I20260806 23:59:22.639461 26856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:22.639607 26856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:22.639698 26856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:22.639796 26856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:59:22.641638 26856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:22.641979 26856 leader_election.cc:302] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 719999eca869445d9815e39b9cfb7fbd; no voters:
I20260806 23:59:22.642344 26856 leader_election.cc:288] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:59:22.642433 26861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:59:22.642674 26861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [term 2 LEADER]: Becoming Leader. State: Replica: 719999eca869445d9815e39b9cfb7fbd, State: Running, Role: LEADER
I20260806 23:59:22.643059 26861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } }
I20260806 23:59:22.643656 26856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:22.644104 26863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "719999eca869445d9815e39b9cfb7fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } } }
I20260806 23:59:22.644227 26863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:22.644472 26862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 719999eca869445d9815e39b9cfb7fbd. Latest consensus state: current_term: 2 leader_uuid: "719999eca869445d9815e39b9cfb7fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719999eca869445d9815e39b9cfb7fbd" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36339 } } }
I20260806 23:59:22.644585 26862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:22.644732 26865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:22.647996 26865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:22.648531 26856 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:59:22.648669 26865 catalog_manager.cc:1259] Loaded cluster ID: e0568150739a42d8b918dcece05df0ee
I20260806 23:59:22.648732 26865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:22.651077 26865 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:22.655423 26865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Loaded TSK: 0
I20260806 23:59:22.657189 26865 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:59:22.757160 26858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:22.757479 26858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:22.757545 26858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:22.763814 26858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:22.763932 26858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:22.764039 26858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:22.772032 26858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:36271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=36553
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:22.773501 26858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:22.774919 26858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:22.788291 26902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.788291 26904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:22.788316 26901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:22.789485 26858 server_base.cc:1034] running on GCE node
I20260806 23:59:22.790143 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:22.791064 26858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:22.792235 26858 hybrid_clock.cc:648] HybridClock initialized: now 1786060762792160 us; error 94 us; skew 500 ppm
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:22.796028 26858 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:22.797194 26858 webserver.cc:460] Webserver started at http://127.8.204.65:36553/ using document root <none> and password file <none>
I20260806 23:59:22.797963 26858 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:22.798064 26858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:22.816785 26858 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 23:59:22.819983 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.821421 26858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:59:22.821563 26858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "d6a20c028f9648e2a6045c7269d14da6"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "5b3c6ba20e2cc0bd4ba280978b59ff97"
server_key_iv: "69566ff5614329bd80eb096d8dc8526d"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:22.822109 26858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:22.840907 26858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:22.844674 26858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:22.844949 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:22.845772 26858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:22.847011 26858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:22.847085 26858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.847146 26858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:22.847227 26858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:22.859239 26858 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:36271
I20260806 23:59:22.859304 27058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:36271 every 8 connection(s)
I20260806 23:59:22.860673 26858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:59:22.867389 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 26858
I20260806 23:59:22.867893 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:36761
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:36339
--webserver_port=43807
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:59:22 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:22.883476 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:22.863694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:52327 (local address 127.8.204.126:36339)
0806 23:59:22.865766 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:22.865786 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:22.865821 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:59:22.866134 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:22.866149 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:22.866565 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:22.866982 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:22.868756 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.869744 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.871460 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.872223 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.872398 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:22.875641 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:22.875648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:22.875669 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:22.875704 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:22.877944 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:22.878648 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:22.878655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:22.878662 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:22.878839 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:22.879283 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:22.879287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:22.879289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:22.879798 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:22.879941 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:22.882189 (+ 2248us) server_negotiation.cc:299] Negotiation successful
0806 23:59:22.882513 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1878,"thread_start_us":122,"threads_started":1}
I20260806 23:59:22.884093 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:22.863823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.65:52327)
0806 23:59:22.864391 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:22.864443 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:22.865264 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:22.867509 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:22.867517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:22.867875 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:22.868551 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:22.868574 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.869905 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.869909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:22.871291 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:22.871307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:22.872646 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:22.872651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:22.872844 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:22.873591 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:22.873616 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:22.875377 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:22.878112 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:22.878118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:22.878133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:22.878492 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:22.878956 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:22.878959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:22.878961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:22.879162 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:22.880013 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:22.880022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:22.880343 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 23:59:22.880565 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 23:59:22.880931 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":122,"threads_started":1}
I20260806 23:59:22.887689 27059 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
I20260806 23:59:22.887954 27059 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:22.888535 27059 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:22.890933 27059 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:36339 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:59:22.891083 27059 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:22.891280 27059 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:22.892186 27059 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:22.892379 27059 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:22.893174 27059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:59:23.025714 27066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:23.026005 27066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:23.026089 27066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:23.030525 27066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:23.030642 27066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:23.030803 27066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:23.036504 27066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:36761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=43807
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:23.037854 27066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:23.039018 27066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:23.049134 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:23.049152 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:23.049261 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:23.049628 27066 server_base.cc:1034] running on GCE node
I20260806 23:59:23.050031 27066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:23.050700 27066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:23.051898 27066 hybrid_clock.cc:648] HybridClock initialized: now 1786060763051868 us; error 45 us; skew 500 ppm
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:23.054860 27066 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:23.055824 27066 webserver.cc:460] Webserver started at http://127.8.204.66:43807/ using document root <none> and password file <none>
I20260806 23:59:23.056408 27066 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:23.056521 27066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:23.079090 27066 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:59:23.082509 27090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.084239 27066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:59:23.084450 27066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "803b60ffcf294917ab6eff8bfd3d1d8e"
format_stamp: "Formatted at 2026-08-06 23:59:21 on dist-test-slave-39l8"
server_key: "a229137f73703e47876da245e7785198"
server_key_iv: "a51513f43bfee20f25210cb2b7371e37"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:23.085299 27066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:23.109016 27066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:23.112969 27066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:23.113195 27066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:23.114058 27066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:23.115247 27066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:23.115315 27066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.115374 27066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:23.115408 27066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.127452 27066 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:36761
I20260806 23:59:23.129418 27243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:36761 every 8 connection(s)
I20260806 23:59:23.130754 27066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:59:23.138587 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27066
I20260806 23:59:23.139120 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:38125
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:36339
--webserver_port=41309
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:23.155239 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:23.135139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:41131 (local address 127.8.204.126:36339)
0806 23:59:23.135277 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:23.135282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:23.135728 (+ 446us) server_negotiation.cc:407] Connection header received
0806 23:59:23.137097 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:23.137101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:23.137165 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:23.137276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:23.141037 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.141894 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.145641 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.146015 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.146116 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:23.149610 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:23.149616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:23.149623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:23.149662 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:23.151687 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.152481 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.152487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.152491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.152620 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.153493 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.153498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.153502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.153792 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:23.153892 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:23.154784 (+ 892us) server_negotiation.cc:299] Negotiation successful
0806 23:59:23.154961 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:59:23.157235 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:23.133997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.66:41131)
0806 23:59:23.135552 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:23.135608 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:23.136826 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:23.137489 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:23.137498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:23.137882 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:23.139131 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:23.139155 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.142086 (+ 2931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.142090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:23.143578 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:23.143592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.146371 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.146375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:23.146621 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:23.147422 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:23.147444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:23.149355 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:23.151850 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:23.151858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:23.151881 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:23.152335 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:23.153110 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:23.153115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:23.153119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:23.153353 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:23.153977 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:23.153985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:23.154418 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 23:59:23.155784 (+ 1366us) client_negotiation.cc:240] Negotiation successful
0806 23:59:23.156158 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":134,"threads_started":1}
I20260806 23:59:23.159016 27245 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
I20260806 23:59:23.159334 27245 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.159952 27245 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:23.161594 27245 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:36339 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:59:23.161772 27245 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.161972 27245 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:23.162794 27245 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.163007 27245 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:23.163874 27245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:59:23.288044 27248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:23.288336 27248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:23.288421 27248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:23.293296 27248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:23.293385 27248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:23.293514 27248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:23.299978 27248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:38125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=41309
--tserver_master_addrs=127.8.204.126:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:23.301354 27248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:23.302670 27248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:23.314769 27260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:23.314864 27257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:23.315124 27256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:23.316249 27248 server_base.cc:1034] running on GCE node
I20260806 23:59:23.316748 27248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:23.317591 27248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:23.318801 27248 hybrid_clock.cc:648] HybridClock initialized: now 1786060763318773 us; error 39 us; skew 500 ppm
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:23.322731 27248 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:23.323927 27248 webserver.cc:460] Webserver started at http://127.8.204.67:41309/ using document root <none> and password file <none>
I20260806 23:59:23.324720 27248 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:23.324869 27248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:23.354967 27248 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:59:23.357949 27269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.359417 27248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:59:23.359608 27248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "0ea9574928594e8ba9eb11de08bb114a"
format_stamp: "Formatted at 2026-08-06 23:59:22 on dist-test-slave-39l8"
server_key: "86776b20f4bab5c54855f513a454ad32"
server_key_iv: "7f813cc10bc86da69025942735337209"
server_key_version: "kuduclusterkey@0"
I20260806 23:59:23.360338 27248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:23.393940 27248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:23.398125 27248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:23.398429 27248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:23.399346 27248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:23.400635 27248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:23.400758 27248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.400887 27248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:23.400965 27248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:23.415920 27248 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:38125
I20260806 23:59:23.417464 27248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:59:23.419586 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:38125 every 8 connection(s)
I20260806 23:59:23.421799 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27248
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:23.445181 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:23.423691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.8.204.126:36339)
0806 23:59:23.425053 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:23.425058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:23.428363 (+ 3305us) server_negotiation.cc:407] Connection header received
0806 23:59:23.428412 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:23.428416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:23.428465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:23.428553 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:23.433226 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.434092 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.435552 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.435957 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.436049 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:23.439282 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:23.439285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:23.439288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:23.439312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:23.440945 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.442022 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.442027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.442029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.442127 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.443136 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.443141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.443143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.443754 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:23.444000 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:23.444825 (+ 825us) server_negotiation.cc:299] Negotiation successful
0806 23:59:23.444957 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":120,"threads_started":1}
I20260806 23:59:23.447872 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:23.423593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:39065 (local address 127.8.204.126:36339)
0806 23:59:23.423974 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:23.423979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:23.424884 (+ 905us) server_negotiation.cc:407] Connection header received
0806 23:59:23.426452 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:23.426457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:23.426520 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:23.426580 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:23.430307 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.431299 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.436925 (+ 5626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.437313 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.437421 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:23.442548 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:23.442552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:23.442555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:23.442581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:23.444384 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.445186 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.445191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.445196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.445289 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:23.446262 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:23.446265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:23.446266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:23.446484 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:23.446563 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:23.447594 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0806 23:59:23.447720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260806 23:59:23.448884 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:23.422263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36339 (local address 127.8.204.67:39065)
0806 23:59:23.424705 (+ 2442us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:23.424768 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:23.426175 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:23.427135 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:23.427144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:23.427532 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:23.430120 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:23.430142 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.431483 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.431486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:23.433039 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:23.433052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:23.437773 (+ 4721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:23.437778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:23.438051 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:23.438829 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:23.438859 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:23.442357 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:23.444578 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:23.444596 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:23.444619 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:23.445064 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:23.445411 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:23.445415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:23.445417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:23.446145 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:23.446834 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:23.446841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:23.447324 (+ 483us) client_negotiation.cc:769] Sending connection context
0806 23:59:23.447539 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 23:59:23.447901 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":196,"threads_started":1}
I20260806 23:59:23.449996 27417 heartbeater.cc:347] Connected to a master server at 127.8.204.126:36339
I20260806 23:59:23.450258 27417 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.450935 27417 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:23.452680 27417 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:36339 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:59:23.452826 27417 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.453001 27417 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:23.453855 27417 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.454047 27417 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
W20260806 23:59:23.454893 27417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 23:59:23 dist-test-slave-39l8 krb5kdc[25387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:23.893427 27059 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:23.893764 27059 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:23.895167 26802 ts_manager.cc:194] Registered new tserver with Master: d6a20c028f9648e2a6045c7269d14da6 (127.8.204.65:36271)
I20260806 23:59:23.896963 26802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:52327
I20260806 23:59:24.164193 27245 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:24.164526 27245 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:24.165454 26802 ts_manager.cc:194] Registered new tserver with Master: 803b60ffcf294917ab6eff8bfd3d1d8e (127.8.204.66:36761)
I20260806 23:59:24.166246 26802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:41131
Aug 06 23:59:24 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:24.455152 27417 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:24.455451 27417 heartbeater.cc:511] Master 127.8.204.126:36339 requested a full tablet report, sending...
I20260806 23:59:24.456270 26800 ts_manager.cc:194] Registered new tserver with Master: 0ea9574928594e8ba9eb11de08bb114a (127.8.204.67:38125)
I20260806 23:59:24.457018 26800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:39065
I20260806 23:59:24.460714 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:59:24.479681 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.462741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.8.204.126:36339)
0806 23:59:24.466905 (+ 4164us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.466910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.467286 (+ 376us) server_negotiation.cc:407] Connection header received
0806 23:59:24.467331 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.467334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.467380 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.467455 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:24.468367 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.469187 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.470362 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.470736 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.470827 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.474048 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:24.474052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:24.474055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:24.474081 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:24.475872 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:24.476416 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:24.476419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:24.476421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:24.476553 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:24.476913 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:24.476917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:24.476919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:24.477155 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:24.477320 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.479386 (+ 2066us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.479511 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4076,"thread_start_us":104,"threads_started":1}
Aug 06 23:59:24 dist-test-slave-39l8 krb5kdc[25387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for HTTP/127.8.204.84@KRBTEST.COM
I20260806 23:59:24.483803 26800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:59:24.488871 26800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:59:24.523023 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.516258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:36271 (local address 127.0.0.1:52558)
0806 23:59:24.517524 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.517536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:24.517623 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:24.518137 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:24.518143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:24.518170 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:24.518323 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.518331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.519866 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.519870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.521323 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.521336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.522416 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.522420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.522539 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.522583 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 23:59:24.522740 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 23:59:24.522866 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":89,"threads_started":1}
I20260806 23:59:24.523718 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.516662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.8.204.65:36271)
0806 23:59:24.517005 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.517013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.517622 (+ 609us) server_negotiation.cc:407] Connection header received
0806 23:59:24.517732 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.517737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.517953 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.518113 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:24.518479 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.519718 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.521460 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.522264 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.522378 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.522532 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.523480 (+ 948us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.523585 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":108,"threads_started":1}
I20260806 23:59:24.527096 26953 tablet_service.cc:1467] Processing CreateTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 (DEFAULT_TABLE table=test-table [id=0ccc39f1644947df876cc0297994e32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:24.529170 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011e152d8c44d44974a8a0ef2ca6307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:24.532637 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.515516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:36761 (local address 127.0.0.1:60432)
0806 23:59:24.517234 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.517251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:24.517346 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:24.527128 (+ 9782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:24.527132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:24.527175 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:24.527286 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.527294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.529023 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.529026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.530179 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.530191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.532234 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.532238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.532334 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.532365 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 23:59:24.532421 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 23:59:24.532489 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":103,"threads_started":1}
I20260806 23:59:24.533457 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.515712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.8.204.66:36761)
0806 23:59:24.526539 (+ 10827us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.526547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.526574 (+ 27us) server_negotiation.cc:407] Connection header received
0806 23:59:24.526655 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.526660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.526961 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.527486 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:24.527647 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.528888 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.530349 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.531195 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.533017 (+ 1822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.533169 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.533223 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.533313 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10695,"thread_start_us":96,"threads_started":1}
I20260806 23:59:24.536788 27150 tablet_service.cc:1467] Processing CreateTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 (DEFAULT_TABLE table=test-table [id=0ccc39f1644947df876cc0297994e32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:24.537987 27150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011e152d8c44d44974a8a0ef2ca6307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:24.543730 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.514124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:38125 (local address 127.0.0.1:53484)
0806 23:59:24.514578 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.514635 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:24.514791 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:24.533454 (+ 18663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:24.533458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:24.533478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:24.533569 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.533575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.540555 (+ 6980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.540559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.541689 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.541702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.543313 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.543317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.543415 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.543443 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:59:24.543493 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:59:24.543566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":138,"threads_started":1}
I20260806 23:59:24.544767 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.515080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.8.204.67:38125)
0806 23:59:24.531496 (+ 16416us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.531503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.531528 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:59:24.531607 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.531613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.531820 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.531997 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:24.539186 (+ 7189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.540392 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.541863 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.542745 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.544319 (+ 1574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.544484 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.544535 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.544630 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16291,"thread_start_us":105,"threads_started":1}
I20260806 23:59:24.548005 27464 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Bootstrap starting.
I20260806 23:59:24.548864 27465 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Bootstrap starting.
I20260806 23:59:24.550841 27331 tablet_service.cc:1467] Processing CreateTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 (DEFAULT_TABLE table=test-table [id=0ccc39f1644947df876cc0297994e32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:24.551460 27465 tablet_bootstrap.cc:654] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:24.552088 27331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8011e152d8c44d44974a8a0ef2ca6307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:24.552649 27465 log.cc:824] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:24.556069 27465 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: No bootstrap required, opened a new log
I20260806 23:59:24.556280 27465 ts_tablet_manager.cc:1397] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 23:59:24.560972 27464 tablet_bootstrap.cc:654] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:24.560758 27465 raft_consensus.cc:348] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.561261 27465 raft_consensus.cc:374] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:24.562009 27465 raft_consensus.cc:729] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a20c028f9648e2a6045c7269d14da6, State: Initialized, Role: FOLLOWER
I20260806 23:59:24.562659 27464 log.cc:824] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:24.562683 27465 consensus_queue.cc:260] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.565181 27465 ts_tablet_manager.cc:1428] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260806 23:59:24.565605 27464 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: No bootstrap required, opened a new log
I20260806 23:59:24.565651 27059 heartbeater.cc:503] Master 127.8.204.126:36339 was elected leader, sending a full tablet report...
I20260806 23:59:24.565840 27464 ts_tablet_manager.cc:1397] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260806 23:59:24.569975 27464 raft_consensus.cc:348] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.570251 27464 raft_consensus.cc:374] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:24.570304 27464 raft_consensus.cc:729] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803b60ffcf294917ab6eff8bfd3d1d8e, State: Initialized, Role: FOLLOWER
I20260806 23:59:24.570971 27464 consensus_queue.cc:260] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.571806 27245 heartbeater.cc:503] Master 127.8.204.126:36339 was elected leader, sending a full tablet report...
I20260806 23:59:24.572187 27464 ts_tablet_manager.cc:1428] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:59:24.573905 27468 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Bootstrap starting.
I20260806 23:59:24.576525 27468 tablet_bootstrap.cc:654] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:24.577420 27468 log.cc:824] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:24.596289 27468 tablet_bootstrap.cc:492] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: No bootstrap required, opened a new log
I20260806 23:59:24.596563 27468 ts_tablet_manager.cc:1397] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20260806 23:59:24.600899 27468 raft_consensus.cc:348] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.601296 27468 raft_consensus.cc:374] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:24.601388 27468 raft_consensus.cc:729] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea9574928594e8ba9eb11de08bb114a, State: Initialized, Role: FOLLOWER
I20260806 23:59:24.601994 27468 consensus_queue.cc:260] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.603276 27468 ts_tablet_manager.cc:1428] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 23:59:24.606920 27417 heartbeater.cc:503] Master 127.8.204.126:36339 was elected leader, sending a full tablet report...
W20260806 23:59:24.614529 27060 tablet.cc:2369] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:59:24.634562 27246 tablet.cc:2369] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:59:24.675446 27419 tablet.cc:2369] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:59:24.730489 27473 raft_consensus.cc:482] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:24.730912 27473 raft_consensus.cc:504] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.732386 27473 leader_election.cc:288] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea9574928594e8ba9eb11de08bb114a (127.8.204.67:38125), d6a20c028f9648e2a6045c7269d14da6 (127.8.204.65:36271)
I20260806 23:59:24.752177 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.733729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:34417 (local address 127.8.204.67:38125)
0806 23:59:24.734043 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.734049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.739072 (+ 5023us) server_negotiation.cc:407] Connection header received
0806 23:59:24.739348 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.739352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.739406 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.739459 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:24.740893 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.741691 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.749511 (+ 7820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.750140 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.750246 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.750301 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.751875 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.751995 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260806 23:59:24.753063 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.732795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:38125 (local address 127.8.204.66:34417)
0806 23:59:24.738930 (+ 6135us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.738949 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:24.739185 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:24.739619 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:24.739622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:24.739642 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:24.739799 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.739806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.741851 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.741854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.745220 (+ 3366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.745232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.750411 (+ 5179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.750414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.750529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.750554 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:59:24.752786 (+ 2232us) client_negotiation.cc:240] Negotiation successful
0806 23:59:24.752876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6042,"thread_start_us":144,"threads_started":1}
I20260806 23:59:24.753583 27463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.733877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:36075 (local address 127.8.204.65:36271)
0806 23:59:24.734153 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.734158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.740128 (+ 5970us) server_negotiation.cc:407] Connection header received
0806 23:59:24.740332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.740335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.740389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.740439 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:24.742825 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.743633 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.750688 (+ 7055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.751355 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.751457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.751505 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.753367 (+ 1862us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.753464 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260806 23:59:24.754145 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.733248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:36271 (local address 127.8.204.66:36075)
0806 23:59:24.740029 (+ 6781us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.740057 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:24.740200 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:24.740598 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:24.740601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:24.740620 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:24.740740 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.740745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.743834 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.743837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.746205 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:24.746217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.751610 (+ 5393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.751614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:24.751719 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.751741 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:59:24.753917 (+ 2176us) client_negotiation.cc:240] Negotiation successful
0806 23:59:24.753996 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6714,"thread_start_us":103,"threads_started":1}
I20260806 23:59:24.754959 27354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011e152d8c44d44974a8a0ef2ca6307" candidate_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea9574928594e8ba9eb11de08bb114a" is_pre_election: true
I20260806 23:59:24.755349 27354 raft_consensus.cc:2454] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803b60ffcf294917ab6eff8bfd3d1d8e in term 0.
I20260806 23:59:24.755882 27002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011e152d8c44d44974a8a0ef2ca6307" candidate_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a20c028f9648e2a6045c7269d14da6" is_pre_election: true
I20260806 23:59:24.755930 27091 leader_election.cc:302] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea9574928594e8ba9eb11de08bb114a, 803b60ffcf294917ab6eff8bfd3d1d8e; no voters:
I20260806 23:59:24.756155 27002 raft_consensus.cc:2454] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803b60ffcf294917ab6eff8bfd3d1d8e in term 0.
I20260806 23:59:24.756309 27473 raft_consensus.cc:2781] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:59:24.756395 27473 raft_consensus.cc:482] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:59:24.756435 27473 raft_consensus.cc:3037] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:24.757819 27473 raft_consensus.cc:504] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.758210 27473 leader_election.cc:288] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [CANDIDATE]: Term 1 election: Requested vote from peers 0ea9574928594e8ba9eb11de08bb114a (127.8.204.67:38125), d6a20c028f9648e2a6045c7269d14da6 (127.8.204.65:36271)
I20260806 23:59:24.759074 27002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011e152d8c44d44974a8a0ef2ca6307" candidate_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a20c028f9648e2a6045c7269d14da6"
I20260806 23:59:24.759197 27002 raft_consensus.cc:3037] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:24.760452 27002 raft_consensus.cc:2454] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803b60ffcf294917ab6eff8bfd3d1d8e in term 1.
I20260806 23:59:24.760989 27092 leader_election.cc:302] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803b60ffcf294917ab6eff8bfd3d1d8e, d6a20c028f9648e2a6045c7269d14da6; no voters:
I20260806 23:59:24.761513 27473 raft_consensus.cc:2781] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:24.761591 27354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8011e152d8c44d44974a8a0ef2ca6307" candidate_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea9574928594e8ba9eb11de08bb114a"
I20260806 23:59:24.761762 27473 raft_consensus.cc:686] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 1 LEADER]: Becoming Leader. State: Replica: 803b60ffcf294917ab6eff8bfd3d1d8e, State: Running, Role: LEADER
I20260806 23:59:24.761734 27354 raft_consensus.cc:3037] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:24.762122 27473 consensus_queue.cc:237] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } }
I20260806 23:59:24.763096 27354 raft_consensus.cc:2454] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803b60ffcf294917ab6eff8bfd3d1d8e in term 1.
I20260806 23:59:24.765645 26800 catalog_manager.cc:5495] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 803b60ffcf294917ab6eff8bfd3d1d8e (127.8.204.66). New cstate: current_term: 1 leader_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "803b60ffcf294917ab6eff8bfd3d1d8e" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 36761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260806 23:59:24.815398 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.811695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.8.204.66:36761)
0806 23:59:24.811849 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:24.811854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:24.811870 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:59:24.811992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:24.811994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:24.812050 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:24.812183 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:59:24.812429 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.812945 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.813820 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:24.814040 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:24.814153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:24.814472 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:59:24.814960 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:59:24.815093 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:24.815173 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 23:59:24.815231 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:59:24.821373 27354 raft_consensus.cc:1261] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 1 FOLLOWER]: Refusing update from remote peer 803b60ffcf294917ab6eff8bfd3d1d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:24.821373 27002 raft_consensus.cc:1261] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 1 FOLLOWER]: Refusing update from remote peer 803b60ffcf294917ab6eff8bfd3d1d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:24.822149 27473 consensus_queue.cc:1035] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a20c028f9648e2a6045c7269d14da6" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:24.822379 27479 consensus_queue.cc:1035] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea9574928594e8ba9eb11de08bb114a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:24.830235 27492 mvcc.cc:205] Tried to move back new op lower bound from 7315704892700475392 to 7315704892469313536. Current Snapshot: MvccSnapshot[applied={T|T < 7315704892700475392}]
I20260806 23:59:24.835870 27497 mvcc.cc:205] Tried to move back new op lower bound from 7315704892700475392 to 7315704892469313536. Current Snapshot: MvccSnapshot[applied={T|T < 7315704892700475392}]
I20260806 23:59:24.842006 26800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:24.842197 26800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:24.844549 26800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 719999eca869445d9815e39b9cfb7fbd: Sending DeleteTablet for 3 replicas of tablet 8011e152d8c44d44974a8a0ef2ca6307
I20260806 23:59:24.845443 26953 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:24 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:52558
I20260806 23:59:24.846753 27150 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:24 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60432
I20260806 23:59:24.846896 27331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8011e152d8c44d44974a8a0ef2ca6307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:24 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53484
I20260806 23:59:24.847512 27507 tablet_replica.cc:330] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6: stopping tablet replica
I20260806 23:59:24.847867 27507 raft_consensus.cc:2229] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:24.848213 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26858
I20260806 23:59:24.848253 27507 raft_consensus.cc:2258] T 8011e152d8c44d44974a8a0ef2ca6307 P d6a20c028f9648e2a6045c7269d14da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:24.854060 27512 tablet_replica.cc:330] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: stopping tablet replica
I20260806 23:59:24.854413 27512 raft_consensus.cc:2229] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:24.854799 27512 raft_consensus.cc:2258] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:24.855186 27511 tablet_replica.cc:330] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: stopping tablet replica
I20260806 23:59:24.855465 27511 raft_consensus.cc:2229] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:59:24.855861 27511 raft_consensus.cc:2258] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:24.857602 27511 ts_tablet_manager.cc:1905] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:24.861431 27511 ts_tablet_manager.cc:1918] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:24.861634 27511 log.cc:1196] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/8011e152d8c44d44974a8a0ef2ca6307
I20260806 23:59:24.862095 27511 ts_tablet_manager.cc:1939] T 8011e152d8c44d44974a8a0ef2ca6307 P 803b60ffcf294917ab6eff8bfd3d1d8e: Deleting consensus metadata
I20260806 23:59:24.863399 27512 ts_tablet_manager.cc:1905] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:24.863557 26778 catalog_manager.cc:4841] TS 803b60ffcf294917ab6eff8bfd3d1d8e (127.8.204.66:36761): tablet 8011e152d8c44d44974a8a0ef2ca6307 (table test-table [id=0ccc39f1644947df876cc0297994e32f]) successfully deleted
I20260806 23:59:24.866022 27512 ts_tablet_manager.cc:1918] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:24.866142 27512 log.cc:1196] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/8011e152d8c44d44974a8a0ef2ca6307
I20260806 23:59:24.866487 27512 ts_tablet_manager.cc:1939] T 8011e152d8c44d44974a8a0ef2ca6307 P 0ea9574928594e8ba9eb11de08bb114a: Deleting consensus metadata
I20260806 23:59:24.867619 26778 catalog_manager.cc:4841] TS 0ea9574928594e8ba9eb11de08bb114a (127.8.204.67:38125): tablet 8011e152d8c44d44974a8a0ef2ca6307 (table test-table [id=0ccc39f1644947df876cc0297994e32f]) successfully deleted
W20260806 23:59:24.867697 26779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:36271 (error 108)
I20260806 23:59:24.868736 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:24.868424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:36271 (local address 127.0.0.1:52572)
0806 23:59:24.868556 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:24.868632 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 23:59:24.868898 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27066
W20260806 23:59:24.869010 26779 catalog_manager.cc:4584] TS d6a20c028f9648e2a6045c7269d14da6 (127.8.204.65:36271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8011e152d8c44d44974a8a0ef2ca6307: Network error: Client connection negotiation failed: client connection to 127.8.204.65:36271: connect: Connection refused (error 111)
I20260806 23:59:24.881603 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27248
I20260806 23:59:24.897672 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 26751
I20260806 23:59:24.909338 9009 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 23:59:25.254980 9009 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 23:59:25.255110 9009 mini_ranger.cc:66] Stopping Ranger...
I20260806 23:59:25.610390 9009 mini_ranger.cc:68] Stopped Ranger
2026-08-06 23:59:25.610 UTC [25427] LOG: received smart shutdown request
2026-08-06 23:59:25.614 UTC [25427] LOG: background worker "logical replication launcher" (PID 25440) exited with exit code 1
2026-08-06 23:59:25.615 UTC [25431] LOG: shutting down
2026-08-06 23:59:25.759 UTC [25427] LOG: database system is shut down
2026-08-06T23:59:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49232 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:59:25 dist-test-slave-39l8 krb5kdc[27529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:59:25 dist-test-slave-39l8 krb5kdc[27529](info): set up 2 sockets
Aug 06 23:59:25 dist-test-slave-39l8 krb5kdc[27529](info): commencing operation
krb5kdc: starting...
W20260806 23:59:27.086920 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:59:27.212026 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37687
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:27.325362 27546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:27.325615 27546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:27.325654 27546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:27.329802 27546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:27.329850 27546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:27.329869 27546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:27.329885 27546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:27.329898 27546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:27.335038 27546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:27.335983 27546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:27.336964 27546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:27.345268 27555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:27.345266 27552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:27.345283 27553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:27.345837 27546 server_base.cc:1034] running on GCE node
I20260806 23:59:27.346441 27546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:27.347615 27546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:27.348775 27546 hybrid_clock.cc:648] HybridClock initialized: now 1786060767348748 us; error 39 us; skew 500 ppm
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:27.351637 27546 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:27.352535 27546 webserver.cc:460] Webserver started at http://127.8.204.126:41005/ using document root <none> and password file <none>
I20260806 23:59:27.353078 27546 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:27.353139 27546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:27.353384 27546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:27.355470 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "a3cba880d1b84c648a15c14b1e270773"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "0ddb0d81930b754e4db94236d8a4b3cf"
server_key_iv: "ac8982a771e5d98cd32dadef42791503"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.356031 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "a3cba880d1b84c648a15c14b1e270773"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "0ddb0d81930b754e4db94236d8a4b3cf"
server_key_iv: "ac8982a771e5d98cd32dadef42791503"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.359586 27546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:59:27.361884 27565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:27.363062 27546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:59:27.363205 27546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "a3cba880d1b84c648a15c14b1e270773"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "0ddb0d81930b754e4db94236d8a4b3cf"
server_key_iv: "ac8982a771e5d98cd32dadef42791503"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.363318 27546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:27.381835 27546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:27.385058 27546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:27.385272 27546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:27.392302 27546 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37687
I20260806 23:59:27.392333 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37687 every 8 connection(s)
I20260806 23:59:27.393220 27546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:27.395948 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:27.398895 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27546
I20260806 23:59:27.399027 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:59:27.399302 9009 external_mini_cluster.cc:1428] Setting key 27f127abb9215f646793681cf28e99e5
I20260806 23:59:27.400053 27636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.125@KRBTEST.COM" created.
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:27.418474 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.402739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37687 (local address 127.0.0.1:51044)
0806 23:59:27.403300 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.403419 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.404810 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.405546 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.405552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.406109 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.406527 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.406551 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.407727 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.407732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.409013 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.409029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.409718 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.409723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.409948 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.410571 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.410607 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.413098 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.415498 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.415507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.415514 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.415739 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.416123 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.416125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.416126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.416231 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.417004 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.417009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.417192 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.417363 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.417621 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
I20260806 23:59:27.418478 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.402739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.8.204.126:37687)
0806 23:59:27.403305 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.403328 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.403544 (+ 216us) server_negotiation.cc:407] Connection header received
0806 23:59:27.405143 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.405147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.405320 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.405478 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.406740 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.407588 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.409152 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.409504 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.409684 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.413277 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.413283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.413293 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.413316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.415384 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.415858 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.415867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.415872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.416018 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.416325 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.416331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.416336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.416876 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.417040 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.417490 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.417700 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":443,"spinlock_wait_cycles":69376,"thread_start_us":162,"threads_started":1}
I20260806 23:59:27.421048 27636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.421844 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.421535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35698)
0806 23:59:27.421681 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.421769 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 23:59:27.422036 27568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
I20260806 23:59:27.422948 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.422690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35712)
0806 23:59:27.422831 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.422879 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 23:59:27.423179 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:34815: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
I20260806 23:59:27.429100 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:34815
--webserver_interface=127.8.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:27.472450 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } attempt: 1
I20260806 23:59:27.473402 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.473046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35726)
0806 23:59:27.473228 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.473303 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 23:59:27.474236 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.473998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35736)
0806 23:59:27.474110 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.474161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 23:59:27.474400 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:34815: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
W20260806 23:59:27.541822 27645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:27.542068 27645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:27.542107 27645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 23:59:27.542585 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } attempt: 2
I20260806 23:59:27.543465 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.543124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35746)
0806 23:59:27.543307 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.543376 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 23:59:27.544179 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.543967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35748)
0806 23:59:27.544070 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.544119 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 23:59:27.544318 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:34815: Network error: Client connection negotiation failed: client connection to 127.8.204.125:34815: connect: Connection refused (error 111)
W20260806 23:59:27.546626 27645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:27.546679 27645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:27.546751 27645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:27.546782 27645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:27.546804 27645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:27.551949 27645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:27.552933 27645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:27.553874 27645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:27.563048 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:27.563154 27645 server_base.cc:1034] running on GCE node
W20260806 23:59:27.563061 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:27.563061 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:27.563849 27645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:27.565044 27645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:27.566262 27645 hybrid_clock.cc:648] HybridClock initialized: now 1786060767566249 us; error 31 us; skew 500 ppm
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:27.569274 27645 init.cc:377] Logged in from keytab as kudu/127.8.204.125@KRBTEST.COM (short username kudu)
I20260806 23:59:27.570163 27645 webserver.cc:460] Webserver started at http://127.8.204.125:41691/ using document root <none> and password file <none>
I20260806 23:59:27.570789 27645 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:27.570883 27645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:27.571131 27645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:27.573170 27645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data/instance:
uuid: "bbeac338f6d349d0a92d34a565e77973"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "753d47a58d287d48468957d06058e68d"
server_key_iv: "f81a0b63096006c00855d95c441c1c79"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.573740 27645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal/instance:
uuid: "bbeac338f6d349d0a92d34a565e77973"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "753d47a58d287d48468957d06058e68d"
server_key_iv: "f81a0b63096006c00855d95c441c1c79"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.577473 27645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 23:59:27.579897 27665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:27.581055 27645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:27.581240 27645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "bbeac338f6d349d0a92d34a565e77973"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "753d47a58d287d48468957d06058e68d"
server_key_iv: "f81a0b63096006c00855d95c441c1c79"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.581357 27645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:27.594980 27645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:27.598369 27645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:27.598599 27645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:27.606410 27645 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.125:34815
I20260806 23:59:27.606469 27739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.125:34815 every 8 connection(s)
I20260806 23:59:27.607438 27645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
I20260806 23:59:27.610231 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:27.613878 27741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.616045 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27645
I20260806 23:59:27.616175 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-1/wal/instance
I20260806 23:59:27.616490 9009 external_mini_cluster.cc:1428] Setting key 5f176d8fa70257626ca37dfa4a72cca7
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.124@KRBTEST.COM" created.
I20260806 23:59:27.628479 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.615773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.8.204.126:37687)
0806 23:59:27.615943 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.615948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.616894 (+ 946us) server_negotiation.cc:407] Connection header received
0806 23:59:27.618241 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.618244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.618293 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.618383 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.619608 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.620465 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.621601 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.622025 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.622167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.624607 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.624613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.624617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.624652 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.626314 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.626893 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.626900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.626905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.627005 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.627328 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.627331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.627332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.627503 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.627577 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.628142 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.628297 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:59:27.629405 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.616188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37687 (local address 127.0.0.1:51054)
0806 23:59:27.616715 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.616776 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.618003 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.618555 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.618566 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.618986 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.619452 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.619465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.620607 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.620610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.621444 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.621453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.622198 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.622201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.622348 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.623017 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.623032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.624434 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.626419 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.626424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.626435 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.626672 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.627093 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.627095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.627096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.627237 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.627591 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.627596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.627884 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.628031 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.628395 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":101,"threads_started":1}
I20260806 23:59:27.630913 27741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.635480 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } attempt: 3
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260806 23:59:27.640594 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.631420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35764)
0806 23:59:27.631520 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.631534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.631609 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.632201 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.632204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.632368 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.632449 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.632454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.633404 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.633408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.633918 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.633925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.634494 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.634498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.634596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.635147 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.635158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.636635 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.638788 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.638791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.638793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.639015 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.639371 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.639374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.639376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.639456 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.639939 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.639942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.640126 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.640262 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.640385 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260806 23:59:27.641021 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.631538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.8.204.125:34815)
0806 23:59:27.631875 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.631884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.631901 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:59:27.631931 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.631935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.632085 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.632231 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.632558 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.633280 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.634033 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.634381 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.634496 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.636795 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.636807 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.636824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.636844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.638606 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.639177 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.639183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.639184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.639259 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.639555 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.639558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.639559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.639820 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.639966 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.640358 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.640481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":257,"thread_start_us":164,"threads_started":1}
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260806 23:59:27.643106 27741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.643898 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.643622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40098)
0806 23:59:27.643770 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.643847 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 23:59:27.644088 27672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
I20260806 23:59:27.645203 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:44179
--webserver_interface=127.8.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:27.645484 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.635884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35766)
0806 23:59:27.636065 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.636078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.636156 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.636648 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.636651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.636812 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.637408 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.637418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.638318 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.638324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.638989 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.638999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.639471 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.639475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.639615 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.640126 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.640141 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.641627 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.643774 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.643777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.643779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.644022 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.644397 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.644403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.644408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.644515 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.644989 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.644991 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.645075 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.645157 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.645267 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 23:59:27.645612 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.635902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.8.204.125:34815)
0806 23:59:27.636378 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.636387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.636400 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:59:27.636432 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.636434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.636505 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.636633 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.637533 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.638218 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.639105 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.639353 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.639467 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.641783 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.641786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.641788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.641810 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.643654 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.644192 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.644195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.644197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.644286 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.644644 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.644648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.644650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.644866 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.644975 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.645251 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.645378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":81,"threads_started":1}
I20260806 23:59:27.647171 27636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.647639 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.647378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40100)
0806 23:59:27.647505 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.647565 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 23:59:27.647857 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.647617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40116)
0806 23:59:27.647731 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.647783 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 23:59:27.647846 27741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
I20260806 23:59:27.648561 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.648237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40126)
0806 23:59:27.648418 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.648481 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 23:59:27.648708 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
I20260806 23:59:27.679888 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 1
I20260806 23:59:27.680814 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.680448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40142)
0806 23:59:27.680637 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.680715 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 23:59:27.681555 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.681329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40154)
0806 23:59:27.681436 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.681485 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 23:59:27.681734 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
I20260806 23:59:27.697127 27741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 1
I20260806 23:59:27.698019 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.697700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40164)
0806 23:59:27.697863 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.697932 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 23:59:27.698824 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.698579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40168)
0806 23:59:27.698690 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.698763 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 23:59:27.698953 27741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
I20260806 23:59:27.756935 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 2
I20260806 23:59:27.757892 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.757544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40170)
0806 23:59:27.757727 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.757799 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 23:59:27.758785 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.758533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40182)
0806 23:59:27.758647 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.758698 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 23:59:27.758911 27636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
W20260806 23:59:27.759665 27752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:27.759902 27752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:27.759940 27752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:27.763886 27752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:27.763935 27752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:27.763955 27752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:27.763971 27752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:27.763986 27752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:27.767109 27741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 2
I20260806 23:59:27.767817 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.767530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40190)
0806 23:59:27.767660 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.767739 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 23:59:27.768478 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.768268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40194)
0806 23:59:27.768376 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.768423 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 23:59:27.768626 27741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:44179: Network error: Client connection negotiation failed: client connection to 127.8.204.124:44179: connect: Connection refused (error 111)
I20260806 23:59:27.769866 27752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37687,127.8.204.125:34815,127.8.204.124:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:27.771009 27752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:27.772008 27752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:27.780184 27762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:27.780191 27758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:27.780205 27759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:27.780361 27752 server_base.cc:1034] running on GCE node
I20260806 23:59:27.781095 27752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:27.782171 27752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:27.783350 27752 hybrid_clock.cc:648] HybridClock initialized: now 1786060767783322 us; error 44 us; skew 500 ppm
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:27.786129 27752 init.cc:377] Logged in from keytab as kudu/127.8.204.124@KRBTEST.COM (short username kudu)
I20260806 23:59:27.787009 27752 webserver.cc:460] Webserver started at http://127.8.204.124:41225/ using document root <none> and password file <none>
I20260806 23:59:27.787554 27752 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:27.787611 27752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:27.787770 27752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:27.789656 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data/instance:
uuid: "ea34acca1d5c4e858999da83ce4209d0"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "de4e309d5b9ba1c83d1ef31cd42278c8"
server_key_iv: "afac75a826f153d71404070f011ad0e1"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.790156 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal/instance:
uuid: "ea34acca1d5c4e858999da83ce4209d0"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "de4e309d5b9ba1c83d1ef31cd42278c8"
server_key_iv: "afac75a826f153d71404070f011ad0e1"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.793728 27752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:59:27.796036 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:27.797199 27752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:27.797360 27752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "ea34acca1d5c4e858999da83ce4209d0"
format_stamp: "Formatted at 2026-08-06 23:59:27 on dist-test-slave-39l8"
server_key: "de4e309d5b9ba1c83d1ef31cd42278c8"
server_key_iv: "afac75a826f153d71404070f011ad0e1"
server_key_version: "encryptionkey@0"
I20260806 23:59:27.797469 27752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:27.826414 27752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:27.829299 27752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:27.829531 27752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:27.836673 27752 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.124:44179
I20260806 23:59:27.836707 27847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.124:44179 every 8 connection(s)
I20260806 23:59:27.837599 27752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
I20260806 23:59:27.840430 27848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:27.842640 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27752
I20260806 23:59:27.842810 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060609240246-9009-0/minicluster-data/master-2/wal/instance
I20260806 23:59:27.843115 9009 external_mini_cluster.cc:1428] Setting key f4641ab771b18be21734d936fe0852e2
I20260806 23:59:27.845062 27848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:27.857230 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.847386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.8.204.126:37687)
0806 23:59:27.847547 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.847552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.847826 (+ 274us) server_negotiation.cc:407] Connection header received
0806 23:59:27.847982 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.847985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.848037 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.848128 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.848789 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.849255 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.850315 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.850612 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.850769 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.853531 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.853537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.853542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.853572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.855160 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.855704 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.855707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.855709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.855821 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.856187 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.856190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.856191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.856415 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.856521 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.856902 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.857047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:59:27.857993 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.847301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37687 (local address 127.0.0.1:51060)
0806 23:59:27.847711 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.847749 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.847882 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.848178 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.848184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.848508 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.848630 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.848636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.849379 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.849383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.850179 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.850196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.850811 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.850816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.850946 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.851440 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.851466 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.853380 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.855259 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.855268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.855279 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.855583 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.855939 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.855942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.855944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.856088 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.856523 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.856527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.856646 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.856839 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.857134 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":149,"threads_started":1}
I20260806 23:59:27.858011 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.844572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.8.204.124:44179)
0806 23:59:27.844993 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.845007 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.845042 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:59:27.845747 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.845766 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.846138 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.846481 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.847404 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.848596 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.849502 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.850000 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.850204 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.852980 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.852985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.852995 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.853017 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.855088 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.855704 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.855710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.855711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.855856 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.856156 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.856162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.856163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.856630 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.856771 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.857064 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.857195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260806 23:59:27.858115 27636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 3
I20260806 23:59:27.859508 27848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.859853 27741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } attempt: 3
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:27.870419 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.859983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:34815 (local address 127.0.0.1:35780)
0806 23:59:27.860065 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.860078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.860142 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.860615 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.860617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.860777 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.860904 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.860912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.861873 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.861876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.862878 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.862889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.863569 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.863667 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.864455 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.864473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.865896 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.868447 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.868451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.868452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.868938 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.869280 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.869284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.869429 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.869960 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.869962 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.870035 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.870135 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.870255 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":46}
I20260806 23:59:27.870419 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.860212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40224)
0806 23:59:27.860333 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.860348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.860420 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.861277 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.861280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.861468 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.861552 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.861557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.863155 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.863934 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.863945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.864543 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.864550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.864664 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.865071 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.865086 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.866528 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.868766 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.868771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.868774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.869032 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.869381 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.869386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.869474 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.869906 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.869909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.870012 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.870126 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.870257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 23:59:27.871011 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.860113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.8.204.125:34815)
0806 23:59:27.860398 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.860402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.860416 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:59:27.860453 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.860456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.860493 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.860569 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.861043 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.861748 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863011 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.863363 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863458 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.866082 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.866086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.866089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.866119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.868331 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869068 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.869072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.869074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.869163 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869535 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.869540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.869544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.869855 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.869978 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.870749 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.870877 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260806 23:59:27.871285 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.858458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40216)
0806 23:59:27.858596 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.858611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.858686 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.859362 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.859367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.859533 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.859636 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.859643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.860957 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.860961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.861750 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.861763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.862407 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.862411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.862491 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.862974 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.862988 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.865201 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.869100 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.869112 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.869443 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.870244 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.870247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.870249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.870339 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.870809 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.870811 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.870900 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.870986 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.871116 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 23:59:27.871673 27848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:59:27.871892 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.858846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.8.204.124:44179)
0806 23:59:27.859109 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.859121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.859134 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:59:27.859162 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.859165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.859212 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.859302 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.859823 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.860831 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.861902 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.862286 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863969 (+ 1683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.865391 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.865395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.865399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.865428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.868967 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.870030 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.870035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.870036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.870128 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.870461 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.870466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.870467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.870665 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.871473 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.871553 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.871666 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260806 23:59:27.871899 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.860883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.8.204.124:44179)
0806 23:59:27.861072 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.861078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.861090 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:59:27.861120 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.861123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.861158 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.861789 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.861874 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.863051 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.864069 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.864406 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.864861 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.866729 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.866735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.866739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.866766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.868620 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869156 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.869163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.869166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.869243 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.869586 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.869590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.869592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.869804 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.869911 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.871625 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.871716 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":130}
I20260806 23:59:27.872704 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.860877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.8.204.126:37687)
0806 23:59:27.860993 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.860999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.861066 (+ 67us) server_negotiation.cc:407] Connection header received
0806 23:59:27.861216 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.861219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.861260 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.861304 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.862110 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.862824 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.863918 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.864234 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.864327 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.867379 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.867383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.867386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.867415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.869618 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.870085 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.870091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.870093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.870170 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.870479 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.870482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.870484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.870671 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.870903 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.872437 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.872555 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:59:27.875443 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: Bootstrap starting.
I20260806 23:59:27.875443 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: Bootstrap starting.
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260806 23:59:27.878295 27741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:27.878309 27636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:27.879317 27741 log.cc:824] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:27.879316 27636 log.cc:824] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:27.880669 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.872131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:44179 (local address 127.0.0.1:40230)
0806 23:59:27.872237 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:27.872252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:27.872326 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:27.872684 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:27.872686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:27.872921 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:27.873018 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.873025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.873971 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.873978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.874683 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:27.874694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.875268 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.875271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:27.875338 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.875847 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:27.875862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:27.877243 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:27.879203 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.879206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.879208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.879442 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.879758 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:27.879761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:27.879762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:27.879860 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:27.880247 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:27.880250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:27.880331 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 23:59:27.880420 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 23:59:27.880554 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
Aug 06 23:59:27 dist-test-slave-39l8 krb5kdc[27529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260806 23:59:27.881201 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.872387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.8.204.124:44179)
0806 23:59:27.872474 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.872480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.872493 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:59:27.872521 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.872523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.872562 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.873124 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.873201 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.873853 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.874865 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.875153 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.875606 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.877377 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.877380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.877383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.877404 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.879083 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.879551 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.879554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.879556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.879655 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.879950 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.879952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.879954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.880138 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.880866 (+ 728us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.880936 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.881043 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 23:59:27.881964 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: No bootstrap required, opened a new log
I20260806 23:59:27.883046 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: No bootstrap required, opened a new log
I20260806 23:59:27.885288 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.874108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.8.204.125:34815)
0806 23:59:27.874215 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.874221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.874429 (+ 208us) server_negotiation.cc:407] Connection header received
0806 23:59:27.874600 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.874603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.874641 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.874686 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.875537 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.876483 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.877539 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.878104 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.878453 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.881251 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.881254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.881258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.881287 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.883333 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.883812 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.883816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.883817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.883900 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.884247 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.884254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.884255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.884481 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.884631 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.885034 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.885157 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:59:27.885311 27848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: Bootstrap starting.
I20260806 23:59:27.887110 27741 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.887334 27741 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:27.887388 27741 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeac338f6d349d0a92d34a565e77973, State: Initialized, Role: FOLLOWER
I20260806 23:59:27.887646 27848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:27.887863 27741 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.888350 27848 log.cc:824] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:27.888396 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:27.888527 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:59:27.888942 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:59:27.889250 27741 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:27.890151 27848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: No bootstrap required, opened a new log
I20260806 23:59:27.891629 27636 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.891945 27636 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:27.892071 27636 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cba880d1b84c648a15c14b1e270773, State: Initialized, Role: FOLLOWER
I20260806 23:59:27.892607 27636 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.894004 27636 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:27.894161 27848 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.894449 27848 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:27.894547 27848 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea34acca1d5c4e858999da83ce4209d0, State: Initialized, Role: FOLLOWER
I20260806 23:59:27.895114 27848 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.896497 27848 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:27.893635 27645 master_runner.cc:186] Error getting master registration for 127.8.204.126:37687: OK, instance_id { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" instance_seqno: 1786060767390299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:59:27.897135 27645 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:59:27.897430 27888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:27.897658 27888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 23:59:27.901285 27906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:59:27.901414 27906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 23:59:27.901875 27919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:59:27.901975 27919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:59:27.903795 27546 master_runner.cc:263] No leader master found from master a3cba880d1b84c648a15c14b1e270773
I20260806 23:59:27.903861 27546 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:59:27.906018 27752 master_runner.cc:263] No leader master found from master ea34acca1d5c4e858999da83ce4209d0
I20260806 23:59:27.906121 27752 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:59:27.911423 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.892589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.8.204.126:37687)
0806 23:59:27.892881 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.892885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.893119 (+ 234us) server_negotiation.cc:407] Connection header received
0806 23:59:27.893312 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.893316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.893364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.893419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.894402 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.895184 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.896620 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.896934 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.897017 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.907645 (+ 10628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.907649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.907652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.907680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.909425 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.909971 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.909975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.909977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.910060 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.910394 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.910397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.910399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.910618 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.910749 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.911123 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.911261 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260806 23:59:27.914548 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.892640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.8.204.124:44179)
0806 23:59:27.892765 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.892769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.905167 (+ 12398us) server_negotiation.cc:407] Connection header received
0806 23:59:27.905367 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.905370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.905412 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.905480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.906355 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.906932 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.908030 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.908341 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.908417 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.910233 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.910237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.910239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.910268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.912071 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.912605 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.912609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.912611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.912700 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.913041 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.913045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.913047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.913534 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.913892 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.914285 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.914400 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:59:27.917680 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:27.892690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.8.204.125:34815)
0806 23:59:27.892787 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:27.892792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:27.897924 (+ 5132us) server_negotiation.cc:407] Connection header received
0806 23:59:27.898116 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:27.898121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:27.898181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:27.898252 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:27.899719 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.900504 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.903852 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:27.904183 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:27.904275 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:27.912845 (+ 8570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:27.912852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:27.912856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:27.912884 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:27.915719 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.916224 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.916230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.916235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.916331 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:27.916675 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:27.916679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:27.916681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:27.916903 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:27.917015 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:27.917377 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 23:59:27.917507 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:59:27.918282 27899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:27.918452 27899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 23:59:27.920711 27882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:59:27.920794 27882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:59:27.966805 27857 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:27.967015 27857 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.968076 27857 leader_election.cc:288] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cba880d1b84c648a15c14b1e270773 (127.8.204.126:37687), ea34acca1d5c4e858999da83ce4209d0 (127.8.204.124:44179)
I20260806 23:59:27.968955 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbeac338f6d349d0a92d34a565e77973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea34acca1d5c4e858999da83ce4209d0" is_pre_election: true
I20260806 23:59:27.969286 27809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeac338f6d349d0a92d34a565e77973 in term 0.
I20260806 23:59:27.969803 27602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbeac338f6d349d0a92d34a565e77973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cba880d1b84c648a15c14b1e270773" is_pre_election: true
I20260806 23:59:27.970081 27602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeac338f6d349d0a92d34a565e77973 in term 0.
I20260806 23:59:27.970793 27672 leader_election.cc:302] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbeac338f6d349d0a92d34a565e77973, ea34acca1d5c4e858999da83ce4209d0; no voters:
I20260806 23:59:27.971850 27857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:59:27.971941 27857 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:59:27.972000 27857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:27.973353 27857 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.973692 27857 leader_election.cc:288] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [CANDIDATE]: Term 1 election: Requested vote from peers a3cba880d1b84c648a15c14b1e270773 (127.8.204.126:37687), ea34acca1d5c4e858999da83ce4209d0 (127.8.204.124:44179)
I20260806 23:59:27.974494 27602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbeac338f6d349d0a92d34a565e77973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cba880d1b84c648a15c14b1e270773"
I20260806 23:59:27.974622 27602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:27.974865 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbeac338f6d349d0a92d34a565e77973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea34acca1d5c4e858999da83ce4209d0"
I20260806 23:59:27.974987 27809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:27.976372 27809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeac338f6d349d0a92d34a565e77973 in term 1.
I20260806 23:59:27.977941 27602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeac338f6d349d0a92d34a565e77973 in term 1.
I20260806 23:59:27.978345 27672 leader_election.cc:302] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cba880d1b84c648a15c14b1e270773, bbeac338f6d349d0a92d34a565e77973; no voters:
I20260806 23:59:27.978658 27857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:27.978940 27857 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [term 1 LEADER]: Becoming Leader. State: Replica: bbeac338f6d349d0a92d34a565e77973, State: Running, Role: LEADER
I20260806 23:59:27.979360 27857 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } }
I20260806 23:59:27.982376 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbeac338f6d349d0a92d34a565e77973. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:27.982558 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:27.983009 27929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:27.984041 27929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:27.991591 27809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [term 1 FOLLOWER]: Refusing update from remote peer bbeac338f6d349d0a92d34a565e77973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:27.994632 27926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:27.998953 27899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbeac338f6d349d0a92d34a565e77973. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:27.999087 27899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:59:28.000283 27602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [term 1 FOLLOWER]: Refusing update from remote peer bbeac338f6d349d0a92d34a565e77973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:28.000916 27926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:28.004982 27888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbeac338f6d349d0a92d34a565e77973. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:28.005103 27888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:59:28.023885 27888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:28.024017 27888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:59:28.024974 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:28.025156 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:28.026469 27930 mvcc.cc:205] Tried to move back new op lower bound from 7315704905685430272 to 7315704905651429376. Current Snapshot: MvccSnapshot[applied={T|T < 7315704905685430272}]
I20260806 23:59:28.027032 27935 mvcc.cc:205] Tried to move back new op lower bound from 7315704905685430272 to 7315704905651429376. Current Snapshot: MvccSnapshot[applied={T|T < 7315704905685430272}]
I20260806 23:59:28.030787 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:28.030913 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973 [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:28.032011 27899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbeac338f6d349d0a92d34a565e77973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cba880d1b84c648a15c14b1e270773" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37687 } } peers { permanent_uuid: "bbeac338f6d349d0a92d34a565e77973" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 34815 } } peers { permanent_uuid: "ea34acca1d5c4e858999da83ce4209d0" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 44179 } } }
I20260806 23:59:28.032131 27899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:59:28.034899 27931 mvcc.cc:205] Tried to move back new op lower bound from 7315704905685430272 to 7315704905651429376. Current Snapshot: MvccSnapshot[applied={T|T < 7315704905685430272}]
I20260806 23:59:28.035429 27929 catalog_manager.cc:1347] Generated new cluster ID: 8eaeaa995ae3414d9f96883974d1d2cc
I20260806 23:59:28.035486 27929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:28.050963 27929 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:59:28.052340 27929 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:28.066511 27929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbeac338f6d349d0a92d34a565e77973: Generated new TSK 0
I20260806 23:59:28.067466 27929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:59:28.903128 27906 catalog_manager.cc:1259] Loaded cluster ID: 8eaeaa995ae3414d9f96883974d1d2cc
I20260806 23:59:28.903204 27919 catalog_manager.cc:1259] Loaded cluster ID: 8eaeaa995ae3414d9f96883974d1d2cc
I20260806 23:59:28.903234 27906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: loading cluster ID for follower catalog manager: success
I20260806 23:59:28.903250 27919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: loading cluster ID for follower catalog manager: success
I20260806 23:59:28.905519 27919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: acquiring CA information for follower catalog manager: success
I20260806 23:59:28.905519 27906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: acquiring CA information for follower catalog manager: success
I20260806 23:59:28.907021 27919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea34acca1d5c4e858999da83ce4209d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:59:28.907021 27906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3cba880d1b84c648a15c14b1e270773: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:59:29.144330 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27546
I20260806 23:59:29.155572 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27645
I20260806 23:59:29.165863 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27752
2026-08-06T23:59:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3289 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:59:29 dist-test-slave-39l8 krb5kdc[27953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:59:29 dist-test-slave-39l8 krb5kdc[27953](info): set up 2 sockets
Aug 06 23:59:29 dist-test-slave-39l8 krb5kdc[27953](info): commencing operation
krb5kdc: starting...
W20260806 23:59:30.376624 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[27953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:59:30.501273 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38715
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:30.613933 27967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:30.614248 27967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:30.614329 27967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:30.618352 27967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:30.618443 27967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:30.618480 27967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:30.618536 27967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:30.618574 27967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:30.623739 27967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:30.624743 27967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:30.625749 27967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:30.634227 27978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:30.634351 27967 server_base.cc:1034] running on GCE node
W20260806 23:59:30.634266 27976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:30.634243 27975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:30.635147 27967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:30.636184 27967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:30.637396 27967 hybrid_clock.cc:648] HybridClock initialized: now 1786060770637378 us; error 31 us; skew 500 ppm
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[27953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060770, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:30.640228 27967 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:30.641120 27967 webserver.cc:460] Webserver started at http://127.8.204.126:35117/ using document root <none> and password file <none>
I20260806 23:59:30.641681 27967 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:30.641785 27967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:30.642015 27967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:30.644104 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "c36d28e8e2d24cdd8ad7706394b7c18a"
format_stamp: "Formatted at 2026-08-06 23:59:30 on dist-test-slave-39l8"
server_key: "bcf9130a3813d0441fc377215995a3b1"
server_key_iv: "8835127ccb6ff92fe5bfd08900cbdffb"
server_key_version: "encryptionkey@0"
I20260806 23:59:30.644676 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "c36d28e8e2d24cdd8ad7706394b7c18a"
format_stamp: "Formatted at 2026-08-06 23:59:30 on dist-test-slave-39l8"
server_key: "bcf9130a3813d0441fc377215995a3b1"
server_key_iv: "8835127ccb6ff92fe5bfd08900cbdffb"
server_key_version: "encryptionkey@0"
I20260806 23:59:30.648376 27967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:59:30.650835 27990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:30.652025 27967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:59:30.652166 27967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "c36d28e8e2d24cdd8ad7706394b7c18a"
format_stamp: "Formatted at 2026-08-06 23:59:30 on dist-test-slave-39l8"
server_key: "bcf9130a3813d0441fc377215995a3b1"
server_key_iv: "8835127ccb6ff92fe5bfd08900cbdffb"
server_key_version: "encryptionkey@0"
I20260806 23:59:30.652281 27967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:30.670934 27967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:30.674036 27967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:30.674245 27967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:30.681772 27967 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:38715
I20260806 23:59:30.681788 28058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:38715 every 8 connection(s)
I20260806 23:59:30.682679 27967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:30.685461 28060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:30.688063 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 27967
I20260806 23:59:30.688199 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:59:30.688469 9009 external_mini_cluster.cc:1428] Setting key 96d339201239fa6e35e95d0b73bf899b
I20260806 23:59:30.691468 28060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a: Bootstrap starting.
I20260806 23:59:30.694041 28060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:30.694959 28060 log.cc:824] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:30.697302 28060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a: No bootstrap required, opened a new log
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[27953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:30.700876 28060 raft_consensus.cc:348] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } }
I20260806 23:59:30.701104 28060 raft_consensus.cc:374] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:30.701185 28060 raft_consensus.cc:729] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36d28e8e2d24cdd8ad7706394b7c18a, State: Initialized, Role: FOLLOWER
I20260806 23:59:30.701628 28060 consensus_queue.cc:260] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } }
I20260806 23:59:30.701784 28060 raft_consensus.cc:388] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:30.701877 28060 raft_consensus.cc:482] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:30.701983 28060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:30.703154 28060 raft_consensus.cc:504] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } }
I20260806 23:59:30.703389 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:30.689752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.8.204.126:38715)
0806 23:59:30.690165 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:30.690180 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:30.690221 (+ 41us) server_negotiation.cc:407] Connection header received
0806 23:59:30.690899 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:30.690918 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:30.691244 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:30.691511 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:30.692519 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:30.693487 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:30.694751 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:30.695289 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:30.695493 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:30.698216 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:30.698221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:30.698235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:30.698262 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:30.700039 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:30.700516 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:30.700522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:30.700527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:30.700642 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:30.700940 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:30.700944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:30.700946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:30.701491 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:30.701712 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:30.702043 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 23:59:30.702355 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":128,"threads_started":1}
I20260806 23:59:30.703528 28060 leader_election.cc:302] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c36d28e8e2d24cdd8ad7706394b7c18a; no voters:
I20260806 23:59:30.703840 28060 leader_election.cc:288] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:59:30.703958 28065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:30.704208 28065 raft_consensus.cc:686] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [term 1 LEADER]: Becoming Leader. State: Replica: c36d28e8e2d24cdd8ad7706394b7c18a, State: Running, Role: LEADER
I20260806 23:59:30.704571 28065 consensus_queue.cc:237] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } }
I20260806 23:59:30.705175 28060 sys_catalog.cc:564] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:30.706187 28068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c36d28e8e2d24cdd8ad7706394b7c18a. Latest consensus state: current_term: 1 leader_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } } }
I20260806 23:59:30.706317 28068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:30.706666 28067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36d28e8e2d24cdd8ad7706394b7c18a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38715 } } }
I20260806 23:59:30.706806 28067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:30.706977 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:59:30.707523 28076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:30.710372 28076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:30.716059 28076 catalog_manager.cc:1347] Generated new cluster ID: 5aed4f62859549119c08bfff9598da86
I20260806 23:59:30.716128 28076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:30.730760 28076 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:59:30.732013 28076 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:30.743170 28076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c36d28e8e2d24cdd8ad7706394b7c18a: Generated new TSK 0
I20260806 23:59:30.744006 28076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:59:30.763302 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 27967
2026-08-06T23:59:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1598 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[28093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[28093](info): set up 2 sockets
Aug 06 23:59:30 dist-test-slave-39l8 krb5kdc[28093](info): commencing operation
krb5kdc: starting...
W20260806 23:59:31.970160 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:59:32.107074 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:32.222527 28109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:32.222877 28109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:32.222977 28109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:32.227094 28109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:32.227198 28109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:32.227236 28109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:32.227274 28109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:32.227309 28109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:32.232764 28109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:32.233956 28109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:32.235028 28109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:32.243592 28115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:32.243578 28116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:32.243561 28118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.243893 28109 server_base.cc:1034] running on GCE node
I20260806 23:59:32.244491 28109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:32.245543 28109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:32.246773 28109 hybrid_clock.cc:648] HybridClock initialized: now 1786060772246753 us; error 34 us; skew 500 ppm
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:32.249619 28109 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:32.250547 28109 webserver.cc:460] Webserver started at http://127.8.204.126:38141/ using document root <none> and password file <none>
I20260806 23:59:32.251209 28109 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:32.251313 28109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:32.251559 28109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:32.253670 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cbd3b4a2992e7517ddad26c44f7f5a95"
server_key_iv: "c82cfb8c78cdae983dbc000f458985f5"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.254273 28109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cbd3b4a2992e7517ddad26c44f7f5a95"
server_key_iv: "c82cfb8c78cdae983dbc000f458985f5"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.258116 28109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:59:32.260656 28126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.261945 28109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:32.262104 28109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cbd3b4a2992e7517ddad26c44f7f5a95"
server_key_iv: "c82cfb8c78cdae983dbc000f458985f5"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.262217 28109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:32.290630 28109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:32.294210 28109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:32.294428 28109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:32.301661 28109 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44645
I20260806 23:59:32.301975 28203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44645 every 8 connection(s)
I20260806 23:59:32.302665 28109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:32.305150 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28109
I20260806 23:59:32.305289 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:59:32.305624 9009 external_mini_cluster.cc:1428] Setting key e1f99e88b3045f3df7870cee655570bf
I20260806 23:59:32.305823 28204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:32.313241 28204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap starting.
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:32.316067 28204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:32.316807 28204 log.cc:824] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:32.318753 28204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: No bootstrap required, opened a new log
I20260806 23:59:32.320755 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:32.307114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.8.204.126:44645)
0806 23:59:32.307544 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:32.307569 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:32.307606 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:59:32.308611 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:32.308631 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:32.308966 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:32.309240 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:32.310199 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.311284 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.312535 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.313000 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.313119 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:32.315883 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:32.315888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:32.315901 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:32.315929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:32.317682 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.318148 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.318154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.318158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.318274 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.318569 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.318573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.318575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.319085 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:32.319310 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:32.319549 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 23:59:32.319873 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260806 23:59:32.323352 28204 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:32.323598 28204 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:32.323669 28204 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Initialized, Role: FOLLOWER
I20260806 23:59:32.324127 28204 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:32.324287 28204 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:32.324368 28204 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:32.324468 28204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:32.325421 28204 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:32.325765 28204 leader_election.cc:302] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a0bd6a5dc648c5adfc0fcece03f90a; no voters:
I20260806 23:59:32.326088 28204 leader_election.cc:288] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:59:32.326222 28210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:32.326509 28210 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 LEADER]: Becoming Leader. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Running, Role: LEADER
I20260806 23:59:32.326865 28210 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:32.327380 28204 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:32.328603 28212 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a0bd6a5dc648c5adfc0fcece03f90a. Latest consensus state: current_term: 1 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:32.328737 28212 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:32.329505 28211 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:32.329687 28211 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:32.329873 28219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:32.332859 28219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:32.338518 28219 catalog_manager.cc:1347] Generated new cluster ID: 73762a94872f46658ff6a53ac01c0998
I20260806 23:59:32.338589 28219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:32.354149 28219 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:59:32.355458 28219 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:32.367560 28219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Generated new TSK 0
I20260806 23:59:32.368276 28219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:59:32.397869 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:32.510867 28233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:32.511188 28233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:32.511262 28233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:32.515381 28233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:32.515460 28233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:32.515547 28233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:32.520673 28233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:32.521740 28233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:32.522809 28233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:32.533025 28242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.533107 28233 server_base.cc:1034] running on GCE node
W20260806 23:59:32.533020 28246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:32.533022 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.533695 28233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:32.534441 28233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:32.535646 28233 hybrid_clock.cc:648] HybridClock initialized: now 1786060772535611 us; error 49 us; skew 500 ppm
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:32.538430 28233 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:32.539355 28233 webserver.cc:460] Webserver started at http://127.8.204.65:39919/ using document root <none> and password file <none>
I20260806 23:59:32.539932 28233 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:32.540023 28233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:32.540264 28233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:32.542297 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cc310df892dca3c2fef52e39f139b761"
server_key_iv: "10d37395d3870e413580b2a0d4cd0bbe"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.542948 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cc310df892dca3c2fef52e39f139b761"
server_key_iv: "10d37395d3870e413580b2a0d4cd0bbe"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.546914 28233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:59:32.549324 28256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.550521 28233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:32.550643 28233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cc310df892dca3c2fef52e39f139b761"
server_key_iv: "10d37395d3870e413580b2a0d4cd0bbe"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.550797 28233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:32.569262 28233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:32.572711 28233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:32.572913 28233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:32.573526 28233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:32.574343 28233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:32.574389 28233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.574460 28233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:32.574492 28233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.583388 28233 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:35931
I20260806 23:59:32.583454 28404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:35931 every 8 connection(s)
I20260806 23:59:32.584316 28233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:59:32.585116 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28233
I20260806 23:59:32.585275 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:59:32.585570 9009 external_mini_cluster.cc:1428] Setting key e61b27d2b8f689e8d4df0413db139d4b
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260806 23:59:32.599463 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:32.586317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:34757 (local address 127.8.204.126:44645)
0806 23:59:32.586460 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:32.586464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:32.587428 (+ 964us) server_negotiation.cc:407] Connection header received
0806 23:59:32.589411 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:32.589414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:32.589512 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:32.589640 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:32.590861 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.591501 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.592655 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.592914 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.593032 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:32.595733 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:32.595736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:32.595739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:32.595772 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:32.597257 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.597783 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.597787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.597789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.597873 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.598228 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.598231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.598233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.598422 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:32.598532 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:32.599201 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 23:59:32.599324 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:59:32.600399 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:32.586651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:34757)
0806 23:59:32.587254 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:32.587312 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:32.589131 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:32.589829 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:32.589836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:32.590180 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:32.590656 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:32.590669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.591633 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.591636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:32.592517 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:32.592527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.593078 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.593082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:32.593261 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:32.594009 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:32.594026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:32.595545 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:32.597383 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:32.597389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:32.597404 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:32.597668 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:32.597968 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:32.597971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:32.597973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:32.598116 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:32.598533 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:32.598540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:32.598875 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 23:59:32.599103 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 23:59:32.599419 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260806 23:59:32.601935 28407 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:32.602223 28407 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:32.602828 28407 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:32.604713 28156 ts_manager.cc:194] Registered new tserver with Master: 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931)
I20260806 23:59:32.606369 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:34757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:59:32.617515 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:32.731784 28413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:32.732036 28413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:32.732084 28413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:32.735999 28413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:32.736052 28413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:32.736110 28413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:32.741117 28413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:32.742036 28413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:32.743042 28413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:32.752318 28423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:32.752318 28427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.752373 28413 server_base.cc:1034] running on GCE node
W20260806 23:59:32.752449 28424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.753014 28413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:32.753710 28413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:32.754930 28413 hybrid_clock.cc:648] HybridClock initialized: now 1786060772754889 us; error 31 us; skew 500 ppm
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:32.757800 28413 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:32.758574 28413 webserver.cc:460] Webserver started at http://127.8.204.66:39609/ using document root <none> and password file <none>
I20260806 23:59:32.759156 28413 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:32.759220 28413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:32.759382 28413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:32.761281 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "d26665b77b38438c850e3916f810b6bb"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "ad117abc37fc792ef188536336a01f5e"
server_key_iv: "de91ba10ab8803cea06e4f3e7d36b6a2"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.761782 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "d26665b77b38438c850e3916f810b6bb"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "ad117abc37fc792ef188536336a01f5e"
server_key_iv: "de91ba10ab8803cea06e4f3e7d36b6a2"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.765337 28413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 23:59:32.767849 28435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.769047 28413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:32.769142 28413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "d26665b77b38438c850e3916f810b6bb"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "ad117abc37fc792ef188536336a01f5e"
server_key_iv: "de91ba10ab8803cea06e4f3e7d36b6a2"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.769225 28413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:32.780516 28413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:32.783154 28413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:32.783301 28413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:32.783834 28413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:32.784610 28413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:32.784657 28413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.784704 28413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:32.784726 28413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.793354 28413 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:46363
I20260806 23:59:32.793426 28583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:46363 every 8 connection(s)
I20260806 23:59:32.794287 28413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:32.804343 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28413
I20260806 23:59:32.804473 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:59:32.804848 9009 external_mini_cluster.cc:1428] Setting key 873b50961dd65304dba279491c8a3574
I20260806 23:59:32.807545 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:32.796072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:33391 (local address 127.8.204.126:44645)
0806 23:59:32.796230 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:32.796234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:32.797109 (+ 875us) server_negotiation.cc:407] Connection header received
0806 23:59:32.798380 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:32.798382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:32.798427 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:32.798503 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:32.799647 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.800168 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.801247 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.801457 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.801531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:32.803883 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:32.803886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:32.803888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:32.803911 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:32.805431 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.805948 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.805952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.805954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.806036 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:32.806410 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:32.806412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:32.806413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:32.806592 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:32.806672 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:32.807328 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 23:59:32.807429 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:59:32.808682 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:32.796479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:33391)
0806 23:59:32.796946 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:32.796996 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:32.798198 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:32.798658 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:32.798664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:32.799053 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:32.799512 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:32.799525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.800287 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.800290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:32.801140 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:32.801149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:32.801595 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:32.801598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:32.801737 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:32.802317 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:32.802332 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:32.803730 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:32.805562 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:32.805568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:32.805582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:32.805840 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:32.806141 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:32.806144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:32.806146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:32.806306 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:32.806762 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:32.806769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:32.807074 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 23:59:32.807284 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 23:59:32.807632 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260806 23:59:32.810098 28584 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:32.810407 28584 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:32.810972 28584 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:32.812142 28156 ts_manager.cc:194] Registered new tserver with Master: d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
I20260806 23:59:32.812786 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:33391
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:59:32.833678 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:32.953174 28594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:32.953536 28594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:32.953624 28594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:32.957862 28594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:32.957949 28594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:32.958037 28594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:32.963348 28594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:32.964395 28594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:32.965461 28594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:32.975538 28605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:32.975589 28604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.975687 28594 server_base.cc:1034] running on GCE node
W20260806 23:59:32.975589 28608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:32.976269 28594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:32.976858 28594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:32.978029 28594 hybrid_clock.cc:648] HybridClock initialized: now 1786060772978011 us; error 28 us; skew 500 ppm
Aug 06 23:59:32 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:32.980870 28594 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:32.981745 28594 webserver.cc:460] Webserver started at http://127.8.204.67:40259/ using document root <none> and password file <none>
I20260806 23:59:32.982331 28594 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:32.982395 28594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:32.982609 28594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:32.984735 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "d6b3221b074244658297d79f5723dc95"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "e549488e29c1fee9a6e6a368f82c6eb3"
server_key_iv: "f571f543250ff22fd4d0ac1d09b3eb65"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.985311 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "d6b3221b074244658297d79f5723dc95"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "e549488e29c1fee9a6e6a368f82c6eb3"
server_key_iv: "f571f543250ff22fd4d0ac1d09b3eb65"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.989053 28594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 23:59:32.991494 28618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:32.992691 28594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:59:32.992820 28594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "d6b3221b074244658297d79f5723dc95"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "e549488e29c1fee9a6e6a368f82c6eb3"
server_key_iv: "f571f543250ff22fd4d0ac1d09b3eb65"
server_key_version: "encryptionkey@0"
I20260806 23:59:32.992921 28594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:33.013240 28594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:33.016215 28594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:33.016423 28594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:33.017036 28594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:33.017884 28594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:33.017933 28594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:33.018003 28594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:33.018042 28594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:33.027254 28594 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:34891
I20260806 23:59:33.027281 28764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:34891 every 8 connection(s)
I20260806 23:59:33.028242 28594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:59:33.030391 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28594
I20260806 23:59:33.030536 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:59:33.030825 9009 external_mini_cluster.cc:1428] Setting key cf6362a403ebd4c38ccc8942d2064499
Aug 06 23:59:33 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:33.042412 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.030162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:38889 (local address 127.8.204.126:44645)
0806 23:59:33.030339 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.030344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.031438 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 23:59:33.033028 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.033031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.033086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.033199 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:33.034344 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.034936 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.036064 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.036292 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.036405 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.038994 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:33.038997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:33.039000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:33.039024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:33.040306 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.040819 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.040824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.040825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.040917 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.041268 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.041272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.041274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.041467 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:33.041582 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.042077 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.042238 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 23:59:33.043404 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.030661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:38889)
0806 23:59:33.031275 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.031320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.032802 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.033374 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.033380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.033735 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:33.034202 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.034215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.035060 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.035064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.035929 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.035939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.036447 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.036452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.036619 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.037227 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:33.037243 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:33.038826 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:33.040414 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.040419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.040432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.040697 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.041020 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.041023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.041026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.041165 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.041594 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:33.041603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:33.041837 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.042034 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.042360 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":150,"threads_started":1}
I20260806 23:59:33.044421 28766 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:33.044677 28766 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:33.045291 28766 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:33.046324 28156 ts_manager.cc:194] Registered new tserver with Master: d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891)
I20260806 23:59:33.047004 28156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:38889
I20260806 23:59:33.056641 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:59:33.066191 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.058639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.8.204.126:44645)
0806 23:59:33.058785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.058789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.058993 (+ 204us) server_negotiation.cc:407] Connection header received
0806 23:59:33.059180 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.059183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.059233 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.059319 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:33.060040 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.060549 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.061407 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.061632 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.061745 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.063204 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:33.063213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:33.063215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:33.063238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:33.064404 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.064852 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.064856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.064857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.064931 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.065240 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.065244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.065246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.065464 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:33.065621 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.065902 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.066028 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:59:33.069037 28156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:59:33.072309 28156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:59:33.088606 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.081610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35931 (local address 127.0.0.1:35920)
0806 23:59:33.082573 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.082588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.082681 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.083679 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.083682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.083708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:33.083810 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.083819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085477 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.085480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.087136 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.087148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.088246 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.088252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.088343 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.088364 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.088436 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.088484 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":190,"thread_start_us":113,"threads_started":1}
I20260806 23:59:33.088676 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.082359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.8.204.65:35931)
0806 23:59:33.083205 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.083220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.083246 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:59:33.083325 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.083330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.083529 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.083703 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:33.083942 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085319 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.087276 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.088104 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.088227 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.088379 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.088465 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.088547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":135,"threads_started":1}
I20260806 23:59:33.088606 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.082314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34891 (local address 127.0.0.1:49632)
0806 23:59:33.083335 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.083348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.083446 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.083877 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.083883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.083905 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:33.083986 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.083991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085477 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.085480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.087137 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.087148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.087964 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.087968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.088074 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.088158 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.088294 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.088430 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":143,"thread_start_us":132,"threads_started":1}
I20260806 23:59:33.089222 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.082772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.8.204.67:34891)
0806 23:59:33.083186 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.083197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.083456 (+ 259us) server_negotiation.cc:407] Connection header received
0806 23:59:33.083543 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.083548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.083734 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.083879 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:33.084121 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085288 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.087296 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.087832 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.087961 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.088106 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.088676 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.088952 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":105,"threads_started":1}
I20260806 23:59:33.089219 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.081662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.8.204.66:46363)
0806 23:59:33.082127 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.082141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.082169 (+ 28us) server_negotiation.cc:407] Connection header received
0806 23:59:33.082388 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.082395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.082612 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.082859 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:33.083190 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085589 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.086962 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.087732 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.087842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.087998 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.088376 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.088482 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":164,"threads_started":1}
I20260806 23:59:33.089617 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.081541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:46363 (local address 127.0.0.1:44232)
0806 23:59:33.081986 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.082035 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.082163 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.082846 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.082852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.082876 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:33.083020 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.083028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.085736 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.085739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.086838 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.086849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.088072 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.088076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.088208 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.088233 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.089394 (+ 1161us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.089475 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260806 23:59:33.091908 28319 tablet_service.cc:1467] Processing CreateTablet for tablet ec483359936241e89a5c6bece688b38c (DEFAULT_TABLE table=test-table [id=9a5965d875994362b9ba9a392901c478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:33.092054 28684 tablet_service.cc:1467] Processing CreateTablet for tablet ec483359936241e89a5c6bece688b38c (DEFAULT_TABLE table=test-table [id=9a5965d875994362b9ba9a392901c478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:33.092212 28499 tablet_service.cc:1467] Processing CreateTablet for tablet ec483359936241e89a5c6bece688b38c (DEFAULT_TABLE table=test-table [id=9a5965d875994362b9ba9a392901c478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:33.092981 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec483359936241e89a5c6bece688b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:33.093118 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec483359936241e89a5c6bece688b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:33.093114 28499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec483359936241e89a5c6bece688b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:33.099459 28790 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap starting.
I20260806 23:59:33.100860 28795 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Bootstrap starting.
I20260806 23:59:33.101279 28791 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Bootstrap starting.
I20260806 23:59:33.102017 28790 tablet_bootstrap.cc:654] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:33.102952 28790 log.cc:824] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:33.103542 28795 tablet_bootstrap.cc:654] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:33.103832 28791 tablet_bootstrap.cc:654] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:33.104471 28795 log.cc:824] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:33.104529 28791 log.cc:824] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:33.105386 28790 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: No bootstrap required, opened a new log
I20260806 23:59:33.105651 28790 ts_tablet_manager.cc:1397] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:59:33.106693 28791 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: No bootstrap required, opened a new log
I20260806 23:59:33.106917 28791 ts_tablet_manager.cc:1397] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:59:33.108013 28795 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: No bootstrap required, opened a new log
I20260806 23:59:33.108203 28795 ts_tablet_manager.cc:1397] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:59:33.110086 28790 raft_consensus.cc:348] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.110381 28790 raft_consensus.cc:374] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:33.110445 28790 raft_consensus.cc:729] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da4b7ef1c6a449481ff5fb21ec1625c, State: Initialized, Role: FOLLOWER
I20260806 23:59:33.110816 28791 raft_consensus.cc:348] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.110951 28790 consensus_queue.cc:260] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.111093 28791 raft_consensus.cc:374] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:33.111152 28791 raft_consensus.cc:729] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26665b77b38438c850e3916f810b6bb, State: Initialized, Role: FOLLOWER
I20260806 23:59:33.111096 28795 raft_consensus.cc:348] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.111436 28795 raft_consensus.cc:374] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:33.111523 28795 raft_consensus.cc:729] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b3221b074244658297d79f5723dc95, State: Initialized, Role: FOLLOWER
I20260806 23:59:33.111570 28791 consensus_queue.cc:260] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.112066 28790 ts_tablet_manager.cc:1428] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:59:33.112031 28795 consensus_queue.cc:260] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.112372 28584 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
I20260806 23:59:33.112416 28407 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
I20260806 23:59:33.112519 28791 ts_tablet_manager.cc:1428] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:59:33.112913 28795 ts_tablet_manager.cc:1428] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:59:33.113199 28766 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
W20260806 23:59:33.279301 28767 tablet.cc:2369] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:59:33.295467 28588 tablet.cc:2369] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:59:33.312620 28800 raft_consensus.cc:482] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:33.312867 28800 raft_consensus.cc:504] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.314131 28800 leader_election.cc:288] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363), d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891)
I20260806 23:59:33.319641 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.315056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:36865 (local address 127.8.204.67:34891)
0806 23:59:33.315209 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.315213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.315327 (+ 114us) server_negotiation.cc:407] Connection header received
0806 23:59:33.315442 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.315445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.315493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.315569 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:33.315901 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.316643 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.318416 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.319034 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.319166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.319224 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.319418 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.319498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 23:59:33.319644 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.314901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34891 (local address 127.8.204.65:36865)
0806 23:59:33.315229 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.315240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.315334 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.315616 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.315621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.315636 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:33.315776 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.315785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.316758 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.316761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.318277 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.318287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.319177 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.319180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.319261 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.319283 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.319382 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.319458 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":132,"thread_start_us":129,"threads_started":1}
I20260806 23:59:33.319813 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.314537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:36101 (local address 127.8.204.66:46363)
0806 23:59:33.314780 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.314785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.315167 (+ 382us) server_negotiation.cc:407] Connection header received
0806 23:59:33.315237 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.315240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.315321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.315413 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:33.315820 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.316637 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.318623 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.319239 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.319344 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.319412 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.319593 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.319676 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 23:59:33.320096 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.314374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:46363 (local address 127.8.204.65:36101)
0806 23:59:33.314837 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.314850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.314997 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.315444 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.315447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.315462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:33.315697 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.315705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.316758 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.316761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.318274 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.318287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.319356 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.319361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.319461 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.319481 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.319553 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.319606 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":150,"thread_start_us":151,"threads_started":1}
I20260806 23:59:33.320456 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec483359936241e89a5c6bece688b38c" candidate_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b3221b074244658297d79f5723dc95" is_pre_election: true
I20260806 23:59:33.320688 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec483359936241e89a5c6bece688b38c" candidate_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26665b77b38438c850e3916f810b6bb" is_pre_election: true
I20260806 23:59:33.320760 28711 raft_consensus.cc:2454] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da4b7ef1c6a449481ff5fb21ec1625c in term 0.
I20260806 23:59:33.320966 28527 raft_consensus.cc:2454] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da4b7ef1c6a449481ff5fb21ec1625c in term 0.
I20260806 23:59:33.321262 28259 leader_election.cc:302] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c, d6b3221b074244658297d79f5723dc95; no voters:
I20260806 23:59:33.321544 28800 raft_consensus.cc:2781] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:59:33.321641 28800 raft_consensus.cc:482] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:59:33.321697 28800 raft_consensus.cc:3037] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:33.322815 28800 raft_consensus.cc:504] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.323238 28800 leader_election.cc:288] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 1 election: Requested vote from peers d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363), d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891)
I20260806 23:59:33.323637 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec483359936241e89a5c6bece688b38c" candidate_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b3221b074244658297d79f5723dc95"
I20260806 23:59:33.323652 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec483359936241e89a5c6bece688b38c" candidate_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26665b77b38438c850e3916f810b6bb"
I20260806 23:59:33.323772 28711 raft_consensus.cc:3037] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:33.323778 28527 raft_consensus.cc:3037] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:33.325186 28527 raft_consensus.cc:2454] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da4b7ef1c6a449481ff5fb21ec1625c in term 1.
I20260806 23:59:33.325299 28711 raft_consensus.cc:2454] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da4b7ef1c6a449481ff5fb21ec1625c in term 1.
I20260806 23:59:33.325598 28259 leader_election.cc:302] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c, d6b3221b074244658297d79f5723dc95; no voters:
I20260806 23:59:33.325778 28800 raft_consensus.cc:2781] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:33.326009 28800 raft_consensus.cc:686] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 LEADER]: Becoming Leader. State: Replica: 5da4b7ef1c6a449481ff5fb21ec1625c, State: Running, Role: LEADER
I20260806 23:59:33.326385 28800 consensus_queue.cc:237] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.329748 28156 catalog_manager.cc:5495] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65). New cstate: current_term: 1 leader_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } health_report { overall_health: UNKNOWN } } }
W20260806 23:59:33.335644 28408 tablet.cc:2369] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:59:33.384006 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.380244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.8.204.65:35931)
0806 23:59:33.380387 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.380393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.380409 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:59:33.380535 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.380537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.380611 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.380735 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:59:33.380983 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.381493 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.382442 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.382662 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.382849 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.383181 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:59:33.383613 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:59:33.383756 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.383798 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.383857 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:59:33.389822 28711 raft_consensus.cc:1261] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Refusing update from remote peer 5da4b7ef1c6a449481ff5fb21ec1625c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:33.389822 28527 raft_consensus.cc:1261] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Refusing update from remote peer 5da4b7ef1c6a449481ff5fb21ec1625c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:33.390565 28800 consensus_queue.cc:1035] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:33.390820 28806 consensus_queue.cc:1035] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:33.398017 28809 mvcc.cc:205] Tried to move back new op lower bound from 7315704927797350400 to 7315704927547645952. Current Snapshot: MvccSnapshot[applied={T|T < 7315704927797350400}]
I20260806 23:59:33.399727 28811 mvcc.cc:205] Tried to move back new op lower bound from 7315704927797350400 to 7315704927547645952. Current Snapshot: MvccSnapshot[applied={T|T < 7315704927797350400}]
I20260806 23:59:33.406589 28156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:33.406793 28156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:33.409111 28156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Sending DeleteTablet for 3 replicas of tablet ec483359936241e89a5c6bece688b38c
I20260806 23:59:33.409876 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet ec483359936241e89a5c6bece688b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:33 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35920
I20260806 23:59:33.410019 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28233
I20260806 23:59:33.410092 28499 tablet_service.cc:1514] Processing DeleteTablet for tablet ec483359936241e89a5c6bece688b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:33 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44232
I20260806 23:59:33.410532 28684 tablet_service.cc:1514] Processing DeleteTablet for tablet ec483359936241e89a5c6bece688b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:33 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:49632
I20260806 23:59:33.416455 28824 tablet_replica.cc:330] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: stopping tablet replica
I20260806 23:59:33.416867 28824 raft_consensus.cc:2229] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:33.417356 28824 raft_consensus.cc:2258] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:33.419646 28822 tablet_replica.cc:330] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: stopping tablet replica
I20260806 23:59:33.420037 28822 raft_consensus.cc:2229] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:33.420389 28822 raft_consensus.cc:2258] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:33.420547 28824 ts_tablet_manager.cc:1905] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:33.421769 28822 ts_tablet_manager.cc:1905] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:33.423621 28824 ts_tablet_manager.cc:1918] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:33.423736 28824 log.cc:1196] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/ec483359936241e89a5c6bece688b38c
I20260806 23:59:33.424100 28824 ts_tablet_manager.cc:1939] T ec483359936241e89a5c6bece688b38c P d6b3221b074244658297d79f5723dc95: Deleting consensus metadata
I20260806 23:59:33.424873 28822 ts_tablet_manager.cc:1918] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:33.425009 28822 log.cc:1196] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/ec483359936241e89a5c6bece688b38c
I20260806 23:59:33.425359 28822 ts_tablet_manager.cc:1939] T ec483359936241e89a5c6bece688b38c P d26665b77b38438c850e3916f810b6bb: Deleting consensus metadata
I20260806 23:59:33.425891 28128 catalog_manager.cc:4841] TS d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): tablet ec483359936241e89a5c6bece688b38c (table test-table [id=9a5965d875994362b9ba9a392901c478]) successfully deleted
I20260806 23:59:33.426695 28128 catalog_manager.cc:4841] TS d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): tablet ec483359936241e89a5c6bece688b38c (table test-table [id=9a5965d875994362b9ba9a392901c478]) successfully deleted
I20260806 23:59:33.427237 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28413
W20260806 23:59:33.427368 28130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:35931 (error 108)
I20260806 23:59:33.429718 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.429430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35931 (local address 127.0.0.1:35928)
0806 23:59:33.429568 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.429642 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 23:59:33.429894 28130 catalog_manager.cc:4584] TS 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec483359936241e89a5c6bece688b38c: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35931: connect: Connection refused (error 111)
I20260806 23:59:33.438329 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28594
I20260806 23:59:33.454870 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28109
I20260806 23:59:33.466981 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--webserver_interface=127.8.204.126
--webserver_port=38141
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:33.580242 28826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:33.580509 28826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:33.580549 28826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:33.584404 28826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:33.584453 28826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:33.584472 28826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:33.584488 28826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:33.584503 28826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:33.589563 28826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:33.590551 28826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:33.591646 28826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:33.607688 28837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:33.607728 28835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:33.607688 28834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:33.608057 28826 server_base.cc:1034] running on GCE node
I20260806 23:59:33.608738 28826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:33.609954 28826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:33.611137 28826 hybrid_clock.cc:648] HybridClock initialized: now 1786060773611108 us; error 55 us; skew 500 ppm
Aug 06 23:59:33 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:33.619235 28826 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:33.620163 28826 webserver.cc:460] Webserver started at http://127.8.204.126:38141/ using document root <none> and password file <none>
I20260806 23:59:33.620780 28826 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:33.620889 28826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:33.625118 28826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:59:33.627521 28844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:33.628731 28826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:59:33.628885 28826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cbd3b4a2992e7517ddad26c44f7f5a95"
server_key_iv: "c82cfb8c78cdae983dbc000f458985f5"
server_key_version: "encryptionkey@0"
I20260806 23:59:33.629490 28826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:33.645256 28826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:33.648204 28826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:33.648406 28826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:33.655408 28826 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44645
I20260806 23:59:33.655718 28909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44645 every 8 connection(s)
I20260806 23:59:33.656488 28826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:33.660979 28911 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:59:33.662596 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap starting.
I20260806 23:59:33.664260 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28826
I20260806 23:59:33.664673 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:35931
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=39919
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:33.680716 28911 log.cc:824] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:33.689759 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:59:33.690191 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap complete.
I20260806 23:59:33.694433 28911 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:33.694881 28911 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Initialized, Role: FOLLOWER
I20260806 23:59:33.695386 28911 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:33.695503 28911 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:33.695601 28911 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:33.695725 28911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:59:33.697683 28911 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:33.698031 28911 leader_election.cc:302] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a0bd6a5dc648c5adfc0fcece03f90a; no voters:
I20260806 23:59:33.698376 28911 leader_election.cc:288] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:59:33.698464 28918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:59:33.698660 28918 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 LEADER]: Becoming Leader. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Running, Role: LEADER
I20260806 23:59:33.698977 28918 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:33.699688 28911 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:33.700321 28919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:33.700450 28919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:33.700692 28919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a0bd6a5dc648c5adfc0fcece03f90a. Latest consensus state: current_term: 2 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:33.700794 28919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:33.701560 28925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:33.706491 28925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:33.707403 28925 catalog_manager.cc:1259] Loaded cluster ID: 73762a94872f46658ff6a53ac01c0998
I20260806 23:59:33.707475 28925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:33.708817 28925 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 23:59:33.713491 28925 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 23:59:33.718905 28925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:33.719655 28925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Loaded TSK: 0
I20260806 23:59:33.720834 28925 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:59:33.785202 28916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:33.785444 28916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:33.785487 28916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:33.789426 28916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:33.789479 28916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:33.789561 28916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:33.794828 28916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:35931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=39919
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:33.795836 28916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:33.796864 28916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:33.807576 28948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:33.807622 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:33.807631 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:33.808070 28916 server_base.cc:1034] running on GCE node
I20260806 23:59:33.808445 28916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:33.808995 28916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:33.810194 28916 hybrid_clock.cc:648] HybridClock initialized: now 1786060773810154 us; error 52 us; skew 500 ppm
Aug 06 23:59:33 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:33.813084 28916 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:33.813993 28916 webserver.cc:460] Webserver started at http://127.8.204.65:39919/ using document root <none> and password file <none>
I20260806 23:59:33.814558 28916 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:33.814627 28916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:33.818569 28916 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:59:33.820760 28961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:33.821987 28916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:33.822080 28916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cc310df892dca3c2fef52e39f139b761"
server_key_iv: "10d37395d3870e413580b2a0d4cd0bbe"
server_key_version: "encryptionkey@0"
I20260806 23:59:33.822656 28916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:33.838274 28916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:33.841511 28916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:33.841717 28916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:33.842353 28916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:33.843514 28972 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:59:33.847563 28916 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:59:33.847625 28916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260806 23:59:33.847705 28916 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:59:33.849417 28916 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:59:33.849478 28916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 23:59:33.849749 28972 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap starting.
I20260806 23:59:33.861011 28916 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:35931
I20260806 23:59:33.861109 29117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:35931 every 8 connection(s)
I20260806 23:59:33.862291 28916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:59:33.870865 28972 log.cc:824] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:33.872548 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 28916
I20260806 23:59:33.872924 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:46363
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=39609
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:59:33 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:33.879460 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.864463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:39609)
0806 23:59:33.865033 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.865075 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.865906 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.867057 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.867065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.867500 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:33.867948 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.867961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.869268 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.869271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.870118 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.870128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.870929 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.870933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.871136 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.871754 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:33.871779 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:33.873642 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:33.876210 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.876217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.876232 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.876560 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.876962 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.876965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.876966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.877152 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.877753 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:33.877760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:33.877994 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.878193 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.878535 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":175,"threads_started":1}
I20260806 23:59:33.879448 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.864452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:39609 (local address 127.8.204.126:44645)
0806 23:59:33.864850 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.864871 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.865210 (+ 339us) server_negotiation.cc:407] Connection header received
0806 23:59:33.866316 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.866326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.866627 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.867001 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:33.868140 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.869093 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.870259 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.870781 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.870969 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.873872 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:33.873879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:33.873895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:33.873956 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:33.876067 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.876683 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.876698 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.876703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.876855 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.877260 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.877265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.877267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.877622 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:33.877765 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.878310 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.878570 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":148,"threads_started":1}
I20260806 23:59:33.880118 28972 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:59:33.880599 28972 tablet_bootstrap.cc:492] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap complete.
I20260806 23:59:33.881752 29118 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:33.882007 29118 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:33.882284 28972 ts_tablet_manager.cc:1397] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20260806 23:59:33.882553 29118 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:33.885196 28866 ts_manager.cc:194] Registered new tserver with Master: 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931)
I20260806 23:59:33.886122 28972 raft_consensus.cc:348] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.886809 28972 raft_consensus.cc:729] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da4b7ef1c6a449481ff5fb21ec1625c, State: Initialized, Role: FOLLOWER
I20260806 23:59:33.887363 28972 consensus_queue.cc:260] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } } peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } }
I20260806 23:59:33.888832 28972 ts_tablet_manager.cc:1428] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20260806 23:59:33.889727 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:39609
I20260806 23:59:33.892406 29118 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
Aug 06 23:59:33 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260806 23:59:33.899056 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.888176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.8.204.65:35931)
0806 23:59:33.888992 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:33.889004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:33.889026 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:59:33.889090 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:33.889096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:33.889280 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:33.889411 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:33.890159 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.891135 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.892061 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.892394 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.892622 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.895455 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:33.895464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:33.895476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:33.895508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:33.897141 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.897631 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.897636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.897640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.897720 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:33.898003 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:33.898008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:33.898010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:33.898237 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:33.898385 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:33.898738 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 23:59:33.898916 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":103,"threads_started":1}
I20260806 23:59:33.899051 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:33.888128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35931 (local address 127.0.0.1:35944)
0806 23:59:33.888497 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:33.888540 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:33.888674 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:33.889450 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:33.889456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:33.889886 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:33.890019 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.890026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.891249 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.891253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.891928 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:33.891938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:33.892617 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:33.892621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:33.892868 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:33.893484 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:33.893499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:33.895314 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:33.897267 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.897276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.897282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.897509 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.897816 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:33.897820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:33.897822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:33.897910 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:33.898357 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:33.898364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:33.898486 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 23:59:33.898628 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 23:59:33.898870 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260806 23:59:33.900434 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet ec483359936241e89a5c6bece688b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:33 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35944
I20260806 23:59:33.901418 29130 tablet_replica.cc:330] stopping tablet replica
I20260806 23:59:33.901716 29130 raft_consensus.cc:2229] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:33.902046 29130 raft_consensus.cc:2258] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:33.903084 29130 ts_tablet_manager.cc:1905] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:33.906519 29130 ts_tablet_manager.cc:1918] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:33.906651 29130 log.cc:1196] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/ec483359936241e89a5c6bece688b38c
I20260806 23:59:33.906965 29130 ts_tablet_manager.cc:1939] T ec483359936241e89a5c6bece688b38c P 5da4b7ef1c6a449481ff5fb21ec1625c: Deleting consensus metadata
I20260806 23:59:33.908560 28848 catalog_manager.cc:4841] TS 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931): tablet ec483359936241e89a5c6bece688b38c (table test-table [id=9a5965d875994362b9ba9a392901c478]) successfully deleted
W20260806 23:59:33.992040 29126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:33.992280 29126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:33.992360 29126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:33.996280 29126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:33.996340 29126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:33.996447 29126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:34.001852 29126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:46363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=39609
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:34.003098 29126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:34.004246 29126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:34.014446 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.014678 29142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.014478 29138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.014991 29126 server_base.cc:1034] running on GCE node
I20260806 23:59:34.015508 29126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:34.016327 29126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:34.017629 29126 hybrid_clock.cc:648] HybridClock initialized: now 1786060774017599 us; error 47 us; skew 500 ppm
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:34.021350 29126 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:34.022310 29126 webserver.cc:460] Webserver started at http://127.8.204.66:39609/ using document root <none> and password file <none>
I20260806 23:59:34.022964 29126 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:34.023061 29126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:34.027130 29126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:59:34.029232 29151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.030455 29126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:34.030594 29126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "d26665b77b38438c850e3916f810b6bb"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "ad117abc37fc792ef188536336a01f5e"
server_key_iv: "de91ba10ab8803cea06e4f3e7d36b6a2"
server_key_version: "encryptionkey@0"
I20260806 23:59:34.031189 29126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:34.048432 29126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:34.051874 29126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:34.052109 29126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:34.052737 29126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:34.053609 29126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:34.053656 29126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.053728 29126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:34.053768 29126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.062952 29126 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:46363
I20260806 23:59:34.063022 29307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:46363 every 8 connection(s)
I20260806 23:59:34.063993 29126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:59:34.072265 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 29126
I20260806 23:59:34.072736 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:34891
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=40259
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:34.078876 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.065830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:34981 (local address 127.8.204.126:44645)
0806 23:59:34.066015 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.066020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.066894 (+ 874us) server_negotiation.cc:407] Connection header received
0806 23:59:34.068195 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.068197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.068250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.068326 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:34.069524 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.070054 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.071243 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.071469 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.071549 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.074641 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:34.074649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:34.074654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:34.074693 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:34.076574 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.077102 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.077109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.077113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.077214 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.077540 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.077546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.077551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.077824 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:34.077935 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.078427 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.078599 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 23:59:34.079613 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.066234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:34981)
0806 23:59:34.066678 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.066773 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.067978 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.068505 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.068512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.068874 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:34.069381 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.069395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.070166 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.070168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.071129 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.071139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.071608 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.071610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.071757 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.072486 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:34.072503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:34.074427 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:34.076722 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:34.076727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:34.076739 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:34.076994 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:34.077312 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:34.077314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:34.077316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:34.077444 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:34.077931 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:34.077936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:34.078161 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.078328 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.078647 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260806 23:59:34.080826 29308 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:34.081081 29308 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:34.081578 29308 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:34.082669 28866 ts_manager.cc:194] Registered new tserver with Master: d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
I20260806 23:59:34.083288 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:34981
W20260806 23:59:34.190589 29314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:34.190904 29314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:34.190981 29314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:34.195087 29314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:34.195142 29314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:34.195227 29314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:34.200551 29314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:34891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=40259
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:34.201774 29314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:34.202872 29314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:34.212438 29324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.212541 29314 server_base.cc:1034] running on GCE node
W20260806 23:59:34.212440 29323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.212455 29326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.213160 29314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:34.213788 29314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:34.214947 29314 hybrid_clock.cc:648] HybridClock initialized: now 1786060774214929 us; error 31 us; skew 500 ppm
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:34.217731 29314 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:34.218521 29314 webserver.cc:460] Webserver started at http://127.8.204.67:40259/ using document root <none> and password file <none>
I20260806 23:59:34.219110 29314 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:34.219182 29314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:34.223008 29314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:59:34.224941 29333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.226171 29314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:34.226305 29314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "d6b3221b074244658297d79f5723dc95"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "e549488e29c1fee9a6e6a368f82c6eb3"
server_key_iv: "f571f543250ff22fd4d0ac1d09b3eb65"
server_key_version: "encryptionkey@0"
I20260806 23:59:34.226933 29314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:34.246096 29314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:34.249368 29314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:34.249578 29314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:34.250186 29314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:34.251048 29314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:34.251096 29314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.251168 29314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:34.251201 29314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.260547 29314 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:34891
I20260806 23:59:34.260576 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:34891 every 8 connection(s)
I20260806 23:59:34.261613 29314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:34.271780 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 29314
I20260806 23:59:34.275461 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.263444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:38363 (local address 127.8.204.126:44645)
0806 23:59:34.263599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.263603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.264835 (+ 1232us) server_negotiation.cc:407] Connection header received
0806 23:59:34.265736 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.265740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.265800 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.265905 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:34.267129 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.267653 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.268798 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.269023 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.269150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.271648 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:34.271651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:34.271653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:34.271677 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:34.273235 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.273893 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.273901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.273902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.273979 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.274331 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.274334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.274336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.274514 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:34.274633 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.275171 (+ 538us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.275292 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:59:34.276330 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.263807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:38363)
0806 23:59:34.264683 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.264725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.265518 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.266086 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.266093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.266432 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:34.266969 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.266983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.267789 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.267793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.268663 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.268673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.269188 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.269192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.269360 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.269977 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:34.269993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:34.271466 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:34.273366 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:34.273372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:34.273387 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:34.273771 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:34.274073 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:34.274077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:34.274079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:34.274225 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:34.274634 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:34.274642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:34.274930 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.275121 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.275434 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":550,"threads_started":1}
I20260806 23:59:34.277511 29503 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44645
I20260806 23:59:34.277787 29503 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:34.278301 29503 heartbeater.cc:511] Master 127.8.204.126:44645 requested a full tablet report, sending...
I20260806 23:59:34.279439 28866 ts_manager.cc:194] Registered new tserver with Master: d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891)
I20260806 23:59:34.280231 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:38363
I20260806 23:59:34.281224 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.272532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.8.204.126:44645)
0806 23:59:34.273451 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.273455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.273469 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:59:34.273506 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.273508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.273552 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.273654 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:34.274374 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.274948 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.276060 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.276295 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.276528 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.277878 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:34.277883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:34.277885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:34.277910 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:34.279510 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.279984 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.279986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.279987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.280058 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:34.280365 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:34.280368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:34.280370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:34.280571 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:34.280773 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.280952 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.281051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":72,"threads_started":1}
I20260806 23:59:34.282909 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:59:34.286633 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.283496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.8.204.126:44645)
0806 23:59:34.283625 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.283630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.283766 (+ 136us) server_negotiation.cc:407] Connection header received
0806 23:59:34.283880 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.283882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.283965 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.284045 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:59:34.284234 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.284693 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.285454 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.285660 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.285735 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.285981 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:59:34.286304 (+ 323us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:59:34.286390 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.286445 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.286489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:59:34.287235 28866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:59:34.290354 28866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:59:34.298223 29028 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12148a6667498b854c2845c759a2f5 (DEFAULT_TABLE table=test-table [id=04aa1fdf568947f0a897a8ab388f1d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:34.298979 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d12148a6667498b854c2845c759a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:34.301892 28972 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap starting.
I20260806 23:59:34.302614 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.297226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:46363 (local address 127.0.0.1:44248)
0806 23:59:34.297455 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.297471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.297564 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.298494 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.298498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.298518 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:34.298607 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.298612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.299923 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.299926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.301032 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.301044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.302259 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.302263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.302355 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.302390 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.302451 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.302505 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260806 23:59:34.302793 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.297384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.8.204.66:46363)
0806 23:59:34.297993 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.298004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.298027 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:59:34.298116 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.298121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.298308 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.298552 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:34.298757 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.299765 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.301198 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.302128 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.302244 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.302394 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.302497 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.302587 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":124,"threads_started":1}
I20260806 23:59:34.302855 28972 tablet_bootstrap.cc:654] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:34.304481 28972 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: No bootstrap required, opened a new log
I20260806 23:59:34.304538 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.296964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34891 (local address 127.0.0.1:49646)
0806 23:59:34.297109 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.297125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.297203 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.298224 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.298228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.298247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:34.298351 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.298358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.299786 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.299790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.300878 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.300890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.304139 (+ 3249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.304142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.304239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.304261 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.304347 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.304428 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 23:59:34.304569 28972 ts_tablet_manager.cc:1397] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 23:59:34.304594 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.297134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.8.204.67:34891)
0806 23:59:34.297766 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.297778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.297806 (+ 28us) server_negotiation.cc:407] Connection header received
0806 23:59:34.297884 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.297889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.298064 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.298233 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:34.298516 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.299422 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.301095 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.304011 (+ 2916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.304137 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.304278 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.304374 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.304460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":180,"threads_started":1}
I20260806 23:59:34.305002 28972 raft_consensus.cc:348] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.305125 28972 raft_consensus.cc:374] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:34.305163 28972 raft_consensus.cc:729] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da4b7ef1c6a449481ff5fb21ec1625c, State: Initialized, Role: FOLLOWER
I20260806 23:59:34.305295 28972 consensus_queue.cc:260] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.305555 28972 ts_tablet_manager.cc:1428] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:59:34.305979 29218 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12148a6667498b854c2845c759a2f5 (DEFAULT_TABLE table=test-table [id=04aa1fdf568947f0a897a8ab388f1d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:34.307232 29218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d12148a6667498b854c2845c759a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:34.308270 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12148a6667498b854c2845c759a2f5 (DEFAULT_TABLE table=test-table [id=04aa1fdf568947f0a897a8ab388f1d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:34.309137 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d12148a6667498b854c2845c759a2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:34.314179 29519 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Bootstrap starting.
I20260806 23:59:34.315229 29522 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Bootstrap starting.
I20260806 23:59:34.317030 29519 tablet_bootstrap.cc:654] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:34.317373 29522 tablet_bootstrap.cc:654] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:34.318033 29519 log.cc:824] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:34.318615 29522 log.cc:824] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:34.320266 29519 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: No bootstrap required, opened a new log
I20260806 23:59:34.320477 29519 ts_tablet_manager.cc:1397] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:59:34.320561 29522 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: No bootstrap required, opened a new log
I20260806 23:59:34.320757 29522 ts_tablet_manager.cc:1397] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 23:59:34.324633 29519 raft_consensus.cc:348] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.324632 29522 raft_consensus.cc:348] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.324946 29519 raft_consensus.cc:374] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:34.324946 29522 raft_consensus.cc:374] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:34.325009 29519 raft_consensus.cc:729] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26665b77b38438c850e3916f810b6bb, State: Initialized, Role: FOLLOWER
I20260806 23:59:34.325009 29522 raft_consensus.cc:729] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b3221b074244658297d79f5723dc95, State: Initialized, Role: FOLLOWER
I20260806 23:59:34.325511 29519 consensus_queue.cc:260] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.325511 29522 consensus_queue.cc:260] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.326282 29308 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
I20260806 23:59:34.326663 29519 ts_tablet_manager.cc:1428] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:59:34.327607 29503 heartbeater.cc:503] Master 127.8.204.126:44645 was elected leader, sending a full tablet report...
I20260806 23:59:34.327665 29522 ts_tablet_manager.cc:1428] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20260806 23:59:34.363698 29120 tablet.cc:2369] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:59:34.424163 29526 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:34.424440 29526 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.425715 29526 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363), 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931)
I20260806 23:59:34.430522 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.426153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:60637 (local address 127.8.204.66:46363)
0806 23:59:34.426306 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.426311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.426356 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:59:34.426421 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.426424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.426481 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.426590 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:34.426918 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.427705 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.429235 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.429845 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.430038 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.430104 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.430332 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.430406 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:59:34.430523 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.425999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:46363 (local address 127.8.204.67:60637)
0806 23:59:34.426135 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.426150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.426284 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.426605 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.426611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.426634 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:34.426800 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.426807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.427828 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.427831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.429098 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.429110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.430043 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.430047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.430165 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.430216 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.430299 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.430381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260806 23:59:34.430730 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.426009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35931 (local address 127.8.204.67:47751)
0806 23:59:34.426515 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.426549 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.426644 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:34.426998 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:34.427000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:34.427018 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:34.427114 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.427118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.428059 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.428062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.429190 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:34.429201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.430161 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.430166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:34.430309 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.430335 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:59:34.430398 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 23:59:34.430474 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":207,"thread_start_us":121,"threads_started":1}
I20260806 23:59:34.431406 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.426132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:47751 (local address 127.8.204.65:35931)
0806 23:59:34.426387 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.426391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.426621 (+ 230us) server_negotiation.cc:407] Connection header received
0806 23:59:34.426785 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.426787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.426842 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.426979 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:34.427227 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.427946 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.429329 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.430000 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.430967 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.431142 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.431207 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.431278 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260806 23:59:34.431401 29248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12148a6667498b854c2845c759a2f5" candidate_uuid: "d6b3221b074244658297d79f5723dc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26665b77b38438c850e3916f810b6bb" is_pre_election: true
I20260806 23:59:34.431727 29248 raft_consensus.cc:2454] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b3221b074244658297d79f5723dc95 in term 0.
I20260806 23:59:34.432044 29061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12148a6667498b854c2845c759a2f5" candidate_uuid: "d6b3221b074244658297d79f5723dc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" is_pre_election: true
I20260806 23:59:34.432307 29334 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95; no voters:
I20260806 23:59:34.432379 29061 raft_consensus.cc:2454] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b3221b074244658297d79f5723dc95 in term 0.
I20260806 23:59:34.432597 29526 raft_consensus.cc:2781] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:59:34.432713 29526 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:59:34.432778 29526 raft_consensus.cc:3037] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:34.433979 29526 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.434456 29526 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [CANDIDATE]: Term 1 election: Requested vote from peers d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363), 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931)
I20260806 23:59:34.434894 29248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12148a6667498b854c2845c759a2f5" candidate_uuid: "d6b3221b074244658297d79f5723dc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26665b77b38438c850e3916f810b6bb"
I20260806 23:59:34.434913 29061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12148a6667498b854c2845c759a2f5" candidate_uuid: "d6b3221b074244658297d79f5723dc95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
I20260806 23:59:34.435035 29248 raft_consensus.cc:3037] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:34.435060 29061 raft_consensus.cc:3037] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:34.436239 29248 raft_consensus.cc:2454] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b3221b074244658297d79f5723dc95 in term 1.
I20260806 23:59:34.436244 29061 raft_consensus.cc:2454] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b3221b074244658297d79f5723dc95 in term 1.
I20260806 23:59:34.436542 29334 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95; no voters:
I20260806 23:59:34.436748 29526 raft_consensus.cc:2781] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:34.436975 29526 raft_consensus.cc:686] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 1 LEADER]: Becoming Leader. State: Replica: d6b3221b074244658297d79f5723dc95, State: Running, Role: LEADER
I20260806 23:59:34.437309 29526 consensus_queue.cc:237] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.440475 28865 catalog_manager.cc:5495] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b3221b074244658297d79f5723dc95 (127.8.204.67). New cstate: current_term: 1 leader_uuid: "d6b3221b074244658297d79f5723dc95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } health_report { overall_health: UNKNOWN } } }
I20260806 23:59:34.468703 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.465056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.8.204.67:34891)
0806 23:59:34.465224 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:34.465229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:34.465246 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:59:34.465350 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:34.465352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:34.465407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:34.465526 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:59:34.465784 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.466303 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.467185 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:34.467407 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:34.467515 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:34.467854 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:59:34.468287 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:59:34.468429 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:34.468470 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 23:59:34.468528 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 23:59:34.475062 29248 raft_consensus.cc:1261] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Refusing update from remote peer d6b3221b074244658297d79f5723dc95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:34.475057 29061 raft_consensus.cc:1261] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Refusing update from remote peer d6b3221b074244658297d79f5723dc95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:34.475816 29532 consensus_queue.cc:1035] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:34.476135 29526 consensus_queue.cc:1035] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:34.491354 28865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:34.491556 28865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:34.494051 28865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Sending DeleteTablet for 3 replicas of tablet 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:34.494814 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d12148a6667498b854c2845c759a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:34 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:35944
I20260806 23:59:34.494872 29218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d12148a6667498b854c2845c759a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:34 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44248
I20260806 23:59:34.494966 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28916
I20260806 23:59:34.501726 29404 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d12148a6667498b854c2845c759a2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:34 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:49646
I20260806 23:59:34.502312 29551 tablet_replica.cc:330] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: stopping tablet replica
I20260806 23:59:34.502444 29550 tablet_replica.cc:330] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: stopping tablet replica
I20260806 23:59:34.502830 29551 raft_consensus.cc:2229] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:59:34.502830 29550 raft_consensus.cc:2229] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:34.503216 29550 raft_consensus.cc:2258] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:34.503332 29551 raft_consensus.cc:2258] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:34.504897 29551 ts_tablet_manager.cc:1905] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:34.505115 29550 ts_tablet_manager.cc:1905] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:34.507910 29551 ts_tablet_manager.cc:1918] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:34.507910 29550 ts_tablet_manager.cc:1918] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:34.508044 29550 log.cc:1196] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/1d12148a6667498b854c2845c759a2f5
I20260806 23:59:34.508172 29551 log.cc:1196] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/1d12148a6667498b854c2845c759a2f5
I20260806 23:59:34.508452 29550 ts_tablet_manager.cc:1939] T 1d12148a6667498b854c2845c759a2f5 P d26665b77b38438c850e3916f810b6bb: Deleting consensus metadata
I20260806 23:59:34.508678 29551 ts_tablet_manager.cc:1939] T 1d12148a6667498b854c2845c759a2f5 P d6b3221b074244658297d79f5723dc95: Deleting consensus metadata
I20260806 23:59:34.509801 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 29126
W20260806 23:59:34.509829 28848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:35931 (error 108)
I20260806 23:59:34.509846 28845 catalog_manager.cc:4841] TS d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): tablet 1d12148a6667498b854c2845c759a2f5 (table test-table [id=04aa1fdf568947f0a897a8ab388f1d45]) successfully deleted
I20260806 23:59:34.514083 28845 catalog_manager.cc:4841] TS d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): tablet 1d12148a6667498b854c2845c759a2f5 (table test-table [id=04aa1fdf568947f0a897a8ab388f1d45]) successfully deleted
I20260806 23:59:34.514986 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:34.514487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35931 (local address 127.0.0.1:35946)
0806 23:59:34.514770 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.514870 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260806 23:59:34.515302 28848 catalog_manager.cc:4584] TS 5da4b7ef1c6a449481ff5fb21ec1625c (127.8.204.65:35931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d12148a6667498b854c2845c759a2f5: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35931: connect: Connection refused (error 111)
I20260806 23:59:34.521966 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 29314
I20260806 23:59:34.534956 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 28826
I20260806 23:59:34.549010 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--webserver_interface=127.8.204.126
--webserver_port=38141
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:34.669531 29554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:34.669786 29554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:34.669826 29554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:34.673835 29554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:34.673897 29554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:34.673916 29554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:34.673969 29554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:34.674001 29554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:34.679404 29554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:34.680492 29554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:34.681525 29554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:34.690112 29564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.690131 29561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.690150 29562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.690193 29554 server_base.cc:1034] running on GCE node
I20260806 23:59:34.690963 29554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:34.692081 29554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:34.693243 29554 hybrid_clock.cc:648] HybridClock initialized: now 1786060774693214 us; error 41 us; skew 500 ppm
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:34.695996 29554 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:34.696939 29554 webserver.cc:460] Webserver started at http://127.8.204.126:38141/ using document root <none> and password file <none>
I20260806 23:59:34.697510 29554 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:34.697603 29554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:34.701403 29554 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:34.703553 29579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.704813 29554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:34.704985 29554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cbd3b4a2992e7517ddad26c44f7f5a95"
server_key_iv: "c82cfb8c78cdae983dbc000f458985f5"
server_key_version: "encryptionkey@0"
I20260806 23:59:34.705571 29554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:34.728722 29554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:34.731719 29554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:34.731923 29554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:34.739538 29554 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44645
I20260806 23:59:34.739849 29653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44645 every 8 connection(s)
I20260806 23:59:34.740641 29554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:34.744941 29654 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:59:34.746548 29654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap starting.
I20260806 23:59:34.746791 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 29554
I20260806 23:59:34.747249 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:35931
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=39919
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:34.764788 29654 log.cc:824] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:34.776364 29654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:59:34.776829 29654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a: Bootstrap complete.
I20260806 23:59:34.780941 29654 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:34.781402 29654 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Initialized, Role: FOLLOWER
I20260806 23:59:34.781978 29654 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:34.782101 29654 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:34.782197 29654 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:34.782321 29654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:59:34.784231 29654 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:34.784565 29654 leader_election.cc:302] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a0bd6a5dc648c5adfc0fcece03f90a; no voters:
I20260806 23:59:34.784937 29654 leader_election.cc:288] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 23:59:34.785022 29662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 23:59:34.785213 29662 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [term 3 LEADER]: Becoming Leader. State: Replica: e6a0bd6a5dc648c5adfc0fcece03f90a, State: Running, Role: LEADER
I20260806 23:59:34.785503 29662 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } }
I20260806 23:59:34.786175 29654 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:34.786428 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a0bd6a5dc648c5adfc0fcece03f90a. Latest consensus state: current_term: 3 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:34.786540 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:34.786952 29668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:34.786985 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a0bd6a5dc648c5adfc0fcece03f90a" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44645 } } }
I20260806 23:59:34.787225 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a0bd6a5dc648c5adfc0fcece03f90a [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:34.793450 29668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:34.794369 29668 catalog_manager.cc:1259] Loaded cluster ID: 73762a94872f46658ff6a53ac01c0998
I20260806 23:59:34.794435 29668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 23:59:34.795591 29668 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee4c9f49ad google::LogMessage::Fail() at ??:0
@ 0x7fee4c9f788a google::LogMessage::SendToLog() at ??:0
@ 0x7fee4c9f44de google::LogMessage::Flush() at ??:0
@ 0x7fee4c9f5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee57337fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fee57338d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fee57335579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fee57379640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee4ec463d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fee4ec46c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fee4ec483f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee4ec37512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee5027b6db start_thread at ??:0
@ 0x7fee4b4f261f clone at ??:0
W20260806 23:59:34.877170 29658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:34.877432 29658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:34.877540 29658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:34.881536 29658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:34.881618 29658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:34.881724 29658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:34.886943 29658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:35931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=39919
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:34.887979 29658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:34.889002 29658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:34.899734 29709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:34.899729 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.899922 29658 server_base.cc:1034] running on GCE node
W20260806 23:59:34.899729 29715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:34.900552 29658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:34.901304 29658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:34.902523 29658 hybrid_clock.cc:648] HybridClock initialized: now 1786060774902481 us; error 48 us; skew 500 ppm
Aug 06 23:59:34 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:34.905946 29658 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:34.907063 29658 webserver.cc:460] Webserver started at http://127.8.204.65:39919/ using document root <none> and password file <none>
I20260806 23:59:34.907773 29658 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:34.907871 29658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:34.912811 29658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 23:59:34.915501 29730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:34.917037 29658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:59:34.917168 29658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "5da4b7ef1c6a449481ff5fb21ec1625c"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "cc310df892dca3c2fef52e39f139b761"
server_key_iv: "10d37395d3870e413580b2a0d4cd0bbe"
server_key_version: "encryptionkey@0"
I20260806 23:59:34.917873 29658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:34.931928 29658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:34.935586 29658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:34.935791 29658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:34.936475 29658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:34.937685 29742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:59:34.941859 29658 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:59:34.941934 29658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260806 23:59:34.942000 29658 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:59:34.944334 29658 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:59:34.944393 29658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:34.944762 29742 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap starting.
I20260806 23:59:34.955929 29658 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:35931
I20260806 23:59:34.956277 29902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:35931 every 8 connection(s)
I20260806 23:59:34.957111 29658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260806 23:59:34.964895 29742 log.cc:824] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:34.965332 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 29658
I20260806 23:59:34.965749 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:46363
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=39609
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:34.972784 29742 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:59:34.973169 29742 tablet_bootstrap.cc:492] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Bootstrap complete.
I20260806 23:59:34.974380 29742 ts_tablet_manager.cc:1397] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.008s
I20260806 23:59:34.976927 29742 raft_consensus.cc:348] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.977844 29742 raft_consensus.cc:729] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da4b7ef1c6a449481ff5fb21ec1625c, State: Initialized, Role: FOLLOWER
I20260806 23:59:34.978381 29742 consensus_queue.cc:260] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:34.979445 29742 ts_tablet_manager.cc:1428] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260806 23:59:35.075524 29909 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:59:34.959595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:57113)
0806 23:59:34.960840 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:34.960899 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:34.961975 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:35.074660 (+112685us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":155,"threads_started":1}
W20260806 23:59:35.076059 29904 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:44645 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:44645: Client connection negotiation failed: client connection to 127.8.204.126:44645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 23:59:35.076884 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.076627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:42849)
0806 23:59:35.076765 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.076829 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 23:59:35.077646 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.077423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:40891)
0806 23:59:35.077536 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.077597 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 23:59:35.077819 29904 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:59:35.099730 29911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:35.100021 29911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:35.100123 29911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:35.104167 29911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:35.104262 29911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:35.104346 29911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:35.109647 29911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:46363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=39609
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:35.110821 29911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:35.111855 29911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:35.121587 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:35.121694 29922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:35.121760 29911 server_base.cc:1034] running on GCE node
W20260806 23:59:35.121670 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:35.122295 29911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:35.123029 29911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:35.124228 29911 hybrid_clock.cc:648] HybridClock initialized: now 1786060775124214 us; error 24 us; skew 500 ppm
Aug 06 23:59:35 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060775, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:35.126914 29911 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:35.127826 29911 webserver.cc:460] Webserver started at http://127.8.204.66:39609/ using document root <none> and password file <none>
I20260806 23:59:35.128404 29911 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:35.128497 29911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:35.132270 29911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260806 23:59:35.134294 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.135485 29911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:59:35.135629 29911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "d26665b77b38438c850e3916f810b6bb"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "ad117abc37fc792ef188536336a01f5e"
server_key_iv: "de91ba10ab8803cea06e4f3e7d36b6a2"
server_key_version: "encryptionkey@0"
I20260806 23:59:35.136164 29911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:35.146212 29911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:35.148984 29911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:35.149196 29911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:35.149785 29911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:35.150637 29911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:35.150717 29911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.150839 29911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:35.150890 29911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.160244 29911 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:46363
I20260806 23:59:35.160265 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:46363 every 8 connection(s)
I20260806 23:59:35.161329 29911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:59:35.164546 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 29911
I20260806 23:59:35.164945 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:34891
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:44645
--webserver_port=40259
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:59:35.165231 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.163385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:58043)
0806 23:59:35.163826 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.163929 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
W20260806 23:59:35.165680 30093 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:44645 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:44645: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
I20260806 23:59:35.166289 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.166087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:41781)
0806 23:59:35.166195 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.166239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 23:59:35.167065 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.166801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:60373)
0806 23:59:35.166935 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.166995 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 23:59:35.167253 30093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:59:35.279623 30096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:35.279872 30096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:35.279943 30096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:35.283893 30096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:35.283943 30096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:35.284034 30096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:35.289079 30096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:34891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=40259
--tserver_master_addrs=127.8.204.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:35.290088 30096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:35.291106 30096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:35.300967 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:35.301024 30101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:35.301129 30096 server_base.cc:1034] running on GCE node
W20260806 23:59:35.300967 30102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:35.301671 30096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:35.302311 30096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:35.303480 30096 hybrid_clock.cc:648] HybridClock initialized: now 1786060775303458 us; error 32 us; skew 500 ppm
Aug 06 23:59:35 dist-test-slave-39l8 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060775, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:35.306167 30096 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:35.307063 30096 webserver.cc:460] Webserver started at http://127.8.204.67:40259/ using document root <none> and password file <none>
I20260806 23:59:35.307623 30096 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:35.307717 30096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:35.311476 30096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:59:35.313510 30115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.314666 30096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:35.314839 30096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "d6b3221b074244658297d79f5723dc95"
format_stamp: "Formatted at 2026-08-06 23:59:32 on dist-test-slave-39l8"
server_key: "e549488e29c1fee9a6e6a368f82c6eb3"
server_key_iv: "f571f543250ff22fd4d0ac1d09b3eb65"
server_key_version: "encryptionkey@0"
I20260806 23:59:35.315411 30096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:35.332172 30096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:35.335258 30096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:35.335410 30096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:35.335950 30096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:35.336745 30096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:35.336792 30096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.336833 30096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:35.336848 30096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:35.346135 30096 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:34891
I20260806 23:59:35.346165 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:34891 every 8 connection(s)
I20260806 23:59:35.347222 30096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:59:35.351227 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.349375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:55015)
0806 23:59:35.349859 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.349967 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
W20260806 23:59:35.351780 30278 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:44645 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:44645: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
I20260806 23:59:35.352481 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.352251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:38013)
0806 23:59:35.352371 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.352422 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 23:59:35.352711 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 30096
I20260806 23:59:35.353477 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:35.353230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:46889)
0806 23:59:35.353354 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:35.353406 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 23:59:35.353734 30278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 23:59:36.079092 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.078455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:39663)
0806 23:59:36.078908 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:36.078989 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":117,"threads_started":1}
I20260806 23:59:36.168201 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.167819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:55769)
0806 23:59:36.168041 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:36.168095 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260806 23:59:36.354804 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.354287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:35053)
0806 23:59:36.354589 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:36.354645 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
I20260806 23:59:36.674530 30290 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:36.674825 30290 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:36.675863 30290 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
Aug 06 23:59:36 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.66@KRBTEST.COM
Aug 06 23:59:36 dist-test-slave-39l8 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.67@KRBTEST.COM
I20260806 23:59:36.689962 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.676717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:48709 (local address 127.8.204.66:46363)
0806 23:59:36.677101 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:36.677111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:36.677142 (+ 31us) server_negotiation.cc:407] Connection header received
0806 23:59:36.677738 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:36.677744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:36.678014 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:36.678292 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:36.679544 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.680484 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.681837 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.682448 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.682624 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:36.685282 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:36.685288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:36.685299 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:36.685325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:36.687352 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:36.687966 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:36.687972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:36.687977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:36.688099 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:36.688485 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:36.688489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:36.688491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:36.688877 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:36.689024 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:36.689531 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 23:59:36.689778 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":91,"threads_started":1}
I20260806 23:59:36.689952 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.676495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:46363 (local address 127.8.204.65:48709)
0806 23:59:36.676784 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:36.676801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:36.676913 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:36.678270 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:36.678277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:36.678882 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:36.679367 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:36.679381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.680628 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.680639 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:36.681691 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:36.681703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.682576 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.682580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:36.682851 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:36.683516 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:36.683532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:36.685055 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:36.687494 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:36.687500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:36.687513 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:36.687843 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:36.688201 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:36.688207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:36.688210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:36.688383 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:36.688999 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:36.689005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:36.689227 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 23:59:36.689419 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 23:59:36.689753 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":231,"spinlock_wait_cycles":31616,"thread_start_us":72,"threads_started":1}
I20260806 23:59:36.689998 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.676136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34891 (local address 127.8.204.65:58717)
0806 23:59:36.676369 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:36.676388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:36.676495 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:36.678268 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:36.678274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:36.678883 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:36.679367 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:36.679383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.680650 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.680655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:36.681685 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:36.681696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.682572 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.682576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:36.682867 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:36.683576 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:36.683595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:36.685443 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:36.687501 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:36.687505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:36.687513 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:36.687851 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:36.688204 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:36.688208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:36.688210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:36.688377 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:36.689024 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:36.689027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:36.689227 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 23:59:36.689419 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 23:59:36.689840 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":261,"spinlock_wait_cycles":41856,"thread_start_us":70,"threads_started":1}
I20260806 23:59:36.689996 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:36.676314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:58717 (local address 127.8.204.67:34891)
0806 23:59:36.676773 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:36.676798 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:36.676835 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:59:36.677543 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:36.677549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:36.677894 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:36.678227 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:36.679546 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.680511 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.681817 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:36.682442 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:36.682625 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:36.685634 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:36.685638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:36.685648 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:36.685676 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:36.687372 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:36.687970 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:36.687978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:36.687988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:36.688099 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:36.688479 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:36.688484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:36.688485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:36.688907 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:36.689056 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:36.689532 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 23:59:36.689809 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":138,"threads_started":1}
W20260806 23:59:36.692731 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:36.692971 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:36.693037 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:36.693302 30290 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:37.080364 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:37.079993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:57351)
0806 23:59:37.080194 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:37.080267 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 23:59:37.169417 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:37.168954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:40575)
0806 23:59:37.169247 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:37.169306 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260806 23:59:37.355952 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:37.355509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:34085)
0806 23:59:37.355799 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:37.355860 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260806 23:59:38.081655 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:38.081196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:33827)
0806 23:59:38.081487 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:38.081551 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
I20260806 23:59:38.170599 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:38.170205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:36189)
0806 23:59:38.170451 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:38.170507 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260806 23:59:38.357326 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:38.356756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:54627)
0806 23:59:38.357151 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:38.357232 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260806 23:59:38.735231 30305 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:38.735386 30305 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:38.735896 30305 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
W20260806 23:59:38.736964 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:38.737161 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:38.737272 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:38.737532 30305 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:39.083012 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:39.082531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:41705)
0806 23:59:39.082851 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:39.082910 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260806 23:59:39.171878 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:39.171401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:58337)
0806 23:59:39.171719 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:39.171779 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260806 23:59:39.358513 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:39.358118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:59663)
0806 23:59:39.358365 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:39.358422 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260806 23:59:40.084551 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:40.084027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:35199)
0806 23:59:40.084368 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:40.084429 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260806 23:59:40.173250 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:40.172803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:60081)
0806 23:59:40.173104 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:40.173157 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":116,"threads_started":1}
I20260806 23:59:40.359654 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:40.359244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:42007)
0806 23:59:40.359506 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:40.359557 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260806 23:59:40.740047 30317 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:40.740245 30317 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:40.740711 30317 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
W20260806 23:59:40.741698 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:40.741856 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:40.741925 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:40.742126 30317 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:41.085811 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:41.085390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:45577)
0806 23:59:41.085656 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:41.085711 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260806 23:59:41.174445 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:41.174004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:50833)
0806 23:59:41.174295 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:41.174351 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260806 23:59:41.360970 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:41.360547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:44977)
0806 23:59:41.360813 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:41.360872 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260806 23:59:42.087114 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:42.086610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:35979)
0806 23:59:42.086936 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:42.086995 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":148,"threads_started":1}
I20260806 23:59:42.175661 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:42.175219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:38269)
0806 23:59:42.175513 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:42.175566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260806 23:59:42.362264 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:42.361749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:43365)
0806 23:59:42.362099 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:42.362160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":79,"threads_started":1}
I20260806 23:59:42.793545 30327 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:42.793682 30327 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:42.794085 30327 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
W20260806 23:59:42.795022 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:42.795135 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:42.795213 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:42.795379 30327 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:43.088456 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:43.087963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:50049)
0806 23:59:43.088301 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:43.088357 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260806 23:59:43.176774 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:43.176377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:44239)
0806 23:59:43.176634 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:43.176683 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260806 23:59:43.363543 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:43.363100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:58937)
0806 23:59:43.363384 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:43.363440 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260806 23:59:44.089932 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:44.089428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:55801)
0806 23:59:44.089755 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:44.089833 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20260806 23:59:44.177906 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:44.177513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:34841)
0806 23:59:44.177757 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:44.177810 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260806 23:59:44.364884 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:44.364364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:43475)
0806 23:59:44.364699 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:44.364767 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260806 23:59:45.091334 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:45.090883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:42129)
0806 23:59:45.091177 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:45.091236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":71,"threads_started":1}
I20260806 23:59:45.179141 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:45.178635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:44233)
0806 23:59:45.178992 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:45.179047 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":92,"threads_started":1}
I20260806 23:59:45.366387 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:45.365938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:46989)
0806 23:59:45.366238 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:45.366293 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":143,"threads_started":1}
I20260806 23:59:46.092576 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:46.092189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:54815)
0806 23:59:46.092426 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:46.092480 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260806 23:59:46.180258 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:46.179867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:59467)
0806 23:59:46.180111 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:46.180160 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260806 23:59:46.367646 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:46.367212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:56483)
0806 23:59:46.367463 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:46.367514 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260806 23:59:47.058832 30347 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:47.059029 30347 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:47.059479 30347 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
W20260806 23:59:47.060269 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:47.060423 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:47.060488 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:47.060698 30347 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:47.093725 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:47.093291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:34913)
0806 23:59:47.093564 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:47.093624 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260806 23:59:47.181448 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:47.181074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:56243)
0806 23:59:47.181310 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:47.181361 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260806 23:59:47.368793 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:47.368420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:54949)
0806 23:59:47.368650 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:47.368703 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260806 23:59:48.095136 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:48.094622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:58299)
0806 23:59:48.094966 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:48.095024 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":172,"threads_started":1}
I20260806 23:59:48.182657 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:48.182270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:48491)
0806 23:59:48.182512 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:48.182565 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260806 23:59:48.370023 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:48.369616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:49885)
0806 23:59:48.369874 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:48.369931 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260806 23:59:49.096377 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:49.095978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:35901)
0806 23:59:49.096226 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:49.096281 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260806 23:59:49.183874 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:49.183484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:36709)
0806 23:59:49.183735 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:49.183787 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260806 23:59:49.248826 30358 raft_consensus.cc:482] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:49.248962 30358 raft_consensus.cc:504] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b3221b074244658297d79f5723dc95" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34891 } } peers { permanent_uuid: "d26665b77b38438c850e3916f810b6bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 46363 } } peers { permanent_uuid: "5da4b7ef1c6a449481ff5fb21ec1625c" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35931 } }
I20260806 23:59:49.249416 30358 leader_election.cc:288] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891), d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363)
W20260806 23:59:49.250284 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d26665b77b38438c850e3916f810b6bb (127.8.204.66:46363): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
W20260806 23:59:49.250452 29731 leader_election.cc:341] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6b3221b074244658297d79f5723dc95 (127.8.204.67:34891): Not found: Tablet not found: 1d12148a6667498b854c2845c759a2f5
I20260806 23:59:49.250537 29731 leader_election.cc:302] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5da4b7ef1c6a449481ff5fb21ec1625c; no voters: d26665b77b38438c850e3916f810b6bb, d6b3221b074244658297d79f5723dc95
I20260806 23:59:49.250690 30358 raft_consensus.cc:2726] T 1d12148a6667498b854c2845c759a2f5 P 5da4b7ef1c6a449481ff5fb21ec1625c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:59:49.371166 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:49.370786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:54361)
0806 23:59:49.371021 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:49.371074 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260806 23:59:50.097755 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:50.097360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.65:53739)
0806 23:59:50.097607 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:50.097658 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260806 23:59:50.185286 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:50.184798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.66:41183)
0806 23:59:50.185140 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:50.185195 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":67,"threads_started":1}
I20260806 23:59:50.353649 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 29658
I20260806 23:59:50.367005 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 29911
I20260806 23:59:50.372509 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:50.372104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44645 (local address 127.8.204.67:46935)
0806 23:59:50.372356 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:50.372418 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260806 23:59:50.376899 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 30096
2026-08-06T23:59:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19612 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:59:50 dist-test-slave-39l8 krb5kdc[30366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:59:50 dist-test-slave-39l8 krb5kdc[30366](info): set up 2 sockets
Aug 06 23:59:50 dist-test-slave-39l8 krb5kdc[30366](info): commencing operation
krb5kdc: starting...
W20260806 23:59:51.596393 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 06 23:59:51 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260806 23:59:51.717280 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:51.826097 30383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:51.826333 30383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:51.826371 30383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:51.830260 30383 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 23:59:51.830305 30383 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 23:59:51.830320 30383 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 23:59:51.830333 30383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:59:51.830350 30383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:51.830367 30383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:51.830382 30383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:59:51.830396 30383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:59:51.835438 30383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:51.836319 30383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:51.837217 30383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:59:51.845664 30383 server_base.cc:1034] running on GCE node
W20260806 23:59:51.845643 30392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:51.845636 30400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:51.845643 30393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:51.846489 30383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:51.847568 30383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:51.848728 30383 hybrid_clock.cc:648] HybridClock initialized: now 1786060791848711 us; error 29 us; skew 500 ppm
Aug 06 23:59:51 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:51.851534 30383 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260806 23:59:51.852368 30383 webserver.cc:460] Webserver started at http://127.8.204.126:34543/ using document root <none> and password file <none>
I20260806 23:59:51.852921 30383 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:51.852981 30383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:51.853156 30383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:51.855129 30383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "cdb1b82a60a4420f88872857c7f77c82"
format_stamp: "Formatted at 2026-08-06 23:59:51 on dist-test-slave-39l8"
server_key: "836fccdd2cd64e3e03cc8354cee22adb"
server_key_iv: "4834fa25aea63ea7761458aac149d2b9"
server_key_version: "encryptionkey@0"
I20260806 23:59:51.855634 30383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "cdb1b82a60a4420f88872857c7f77c82"
format_stamp: "Formatted at 2026-08-06 23:59:51 on dist-test-slave-39l8"
server_key: "836fccdd2cd64e3e03cc8354cee22adb"
server_key_iv: "4834fa25aea63ea7761458aac149d2b9"
server_key_version: "encryptionkey@0"
I20260806 23:59:51.859205 30383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:59:51.861433 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:51.862686 30383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:51.862881 30383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "cdb1b82a60a4420f88872857c7f77c82"
format_stamp: "Formatted at 2026-08-06 23:59:51 on dist-test-slave-39l8"
server_key: "836fccdd2cd64e3e03cc8354cee22adb"
server_key_iv: "4834fa25aea63ea7761458aac149d2b9"
server_key_version: "encryptionkey@0"
I20260806 23:59:51.862990 30383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:51.873858 30383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:51.876997 30383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:51.877199 30383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:51.884040 30383 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37545
I20260806 23:59:51.884327 30488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37545 every 8 connection(s)
I20260806 23:59:51.884994 30383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260806 23:59:51.887614 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:51.892904 30489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap starting.
I20260806 23:59:51.893239 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 30383
I20260806 23:59:51.893379 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260806 23:59:51.893669 9009 external_mini_cluster.cc:1428] Setting key a945e6f706fc641429e6a97ee4c800f1
I20260806 23:59:51.895466 30489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:51.896186 30489 log.cc:824] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:51.898290 30489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: No bootstrap required, opened a new log
I20260806 23:59:51.902447 30489 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
Aug 06 23:59:51 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:51.902684 30489 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:51.902791 30489 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Initialized, Role: FOLLOWER
I20260806 23:59:51.903311 30489 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260806 23:59:51.903455 30489 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:59:51.903533 30489 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:59:51.903632 30489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:51.904479 30489 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260806 23:59:51.904851 30489 leader_election.cc:302] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb1b82a60a4420f88872857c7f77c82; no voters:
I20260806 23:59:51.905165 30489 leader_election.cc:288] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:59:51.905278 30495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:51.905488 30495 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 LEADER]: Becoming Leader. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Running, Role: LEADER
I20260806 23:59:51.905874 30495 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260806 23:59:51.906471 30489 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:59:51.907130 30498 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb1b82a60a4420f88872857c7f77c82. Latest consensus state: current_term: 1 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260806 23:59:51.907336 30498 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:51.907639 30497 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260806 23:59:51.907745 30497 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260806 23:59:51.907788 30510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:59:51.908803 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:51.894985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.8.204.126:37545)
0806 23:59:51.895405 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:51.895421 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:51.895456 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:59:51.896115 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:51.896131 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:51.896491 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:51.896764 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:51.897540 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:51.898968 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:51.899813 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:51.900561 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:51.900752 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:51.903356 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:51.903361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:51.903372 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:51.903397 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:51.905435 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:51.905980 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:51.905987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:51.905993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:51.906123 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:51.906427 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:51.906430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:51.906431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:51.906950 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:51.907161 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:51.907597 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 23:59:51.907899 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260806 23:59:51.910861 30510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:59:51.916093 30510 catalog_manager.cc:1347] Generated new cluster ID: a452088a89fb49ad85ce9f07435bb8bd
I20260806 23:59:51.916158 30510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:59:51.961349 30510 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:59:51.962635 30510 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:59:51.978920 30510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Generated new TSK 0
I20260806 23:59:51.979697 30510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260806 23:59:52.010830 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:52.135056 30528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:52.135363 30528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:52.135447 30528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:52.139489 30528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:52.139565 30528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:52.139652 30528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260806 23:59:52.144564 30528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:52.145586 30528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:52.146682 30528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:52.156492 30534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:52.156477 30535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:52.156545 30541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:52.156896 30528 server_base.cc:1034] running on GCE node
I20260806 23:59:52.157334 30528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:52.157948 30528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:52.159138 30528 hybrid_clock.cc:648] HybridClock initialized: now 1786060792159126 us; error 26 us; skew 500 ppm
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:52.162030 30528 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260806 23:59:52.163028 30528 webserver.cc:460] Webserver started at http://127.8.204.65:34227/ using document root <none> and password file <none>
I20260806 23:59:52.163607 30528 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:52.163667 30528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:52.163920 30528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:52.166067 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "ac9671421c00422ba185d0947ab6f97e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "81f965793886b36a36c188235ad8b5ab"
server_key_iv: "41bd90bf2927c66ae3e68313844369da"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.166653 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "ac9671421c00422ba185d0947ab6f97e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "81f965793886b36a36c188235ad8b5ab"
server_key_iv: "41bd90bf2927c66ae3e68313844369da"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.170408 30528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:59:52.172916 30552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.174166 30528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:59:52.174273 30528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "ac9671421c00422ba185d0947ab6f97e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "81f965793886b36a36c188235ad8b5ab"
server_key_iv: "41bd90bf2927c66ae3e68313844369da"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.174355 30528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:52.193651 30528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:52.196826 30528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:52.196991 30528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:52.197585 30528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:52.198442 30528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:52.198489 30528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.198571 30528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:52.198603 30528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.207681 30528 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:33789
I20260806 23:59:52.207722 30712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:33789 every 8 connection(s)
I20260806 23:59:52.208590 30528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:52.217599 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 30528
I20260806 23:59:52.217741 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260806 23:59:52.218040 9009 external_mini_cluster.cc:1428] Setting key abd34f5312ac99401ceba20970f29f81
I20260806 23:59:52.222014 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.210284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:49163 (local address 127.8.204.126:37545)
0806 23:59:52.210450 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.210455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.211535 (+ 1080us) server_negotiation.cc:407] Connection header received
0806 23:59:52.212385 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.212387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.212432 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.212507 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:52.213599 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.214105 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.215310 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.215536 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.215617 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.218059 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:52.218062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:52.218064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:52.218087 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:52.219827 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.220332 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.220338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.220343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.220452 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.220783 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.220789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.220794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.221050 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:52.221162 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.221619 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.221786 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 23:59:52.222944 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.210668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:49163)
0806 23:59:52.211402 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.211441 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.212210 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.212671 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.212677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.213013 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:52.213475 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.213487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.214207 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.214210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.215201 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.215211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.215669 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.215671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.215808 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.216360 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:52.216376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:52.217895 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:52.219956 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.219962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.219974 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.220220 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.220553 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.220556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.220557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.220683 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.221157 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:52.221162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:52.221378 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.221529 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.221850 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":110,"threads_started":1}
I20260806 23:59:52.224222 30713 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260806 23:59:52.224465 30713 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:52.224933 30713 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260806 23:59:52.226740 30443 ts_manager.cc:194] Registered new tserver with Master: ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789)
I20260806 23:59:52.228453 30443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:49163
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260806 23:59:52.247138 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:52.358243 30718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:52.358500 30718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:52.358544 30718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:52.362491 30718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:52.362542 30718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:52.362598 30718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260806 23:59:52.367749 30718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:52.368713 30718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:52.369690 30718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:52.379302 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:52.379320 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:52.379384 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:52.379740 30718 server_base.cc:1034] running on GCE node
I20260806 23:59:52.380146 30718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:52.380746 30718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:52.381934 30718 hybrid_clock.cc:648] HybridClock initialized: now 1786060792381920 us; error 27 us; skew 500 ppm
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:52.384606 30718 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260806 23:59:52.385466 30718 webserver.cc:460] Webserver started at http://127.8.204.66:37163/ using document root <none> and password file <none>
I20260806 23:59:52.386015 30718 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:52.386121 30718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:52.386341 30718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:52.388370 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "c61b77292d074595971a8f922a4691d6"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "e846cbd346250eb96eabd065c4c6d345"
server_key_iv: "57f0ea2f7c9f1388537e63c1bbec6eb5"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.388929 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "c61b77292d074595971a8f922a4691d6"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "e846cbd346250eb96eabd065c4c6d345"
server_key_iv: "57f0ea2f7c9f1388537e63c1bbec6eb5"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.393143 30718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 23:59:52.395835 30737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.397027 30718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:59:52.397183 30718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "c61b77292d074595971a8f922a4691d6"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "e846cbd346250eb96eabd065c4c6d345"
server_key_iv: "57f0ea2f7c9f1388537e63c1bbec6eb5"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.397296 30718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:52.414574 30718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:52.417641 30718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:52.417858 30718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:52.418449 30718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:52.419342 30718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:52.419414 30718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.419485 30718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:52.419535 30718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.428428 30718 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:37889
I20260806 23:59:52.428453 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:37889 every 8 connection(s)
I20260806 23:59:52.429294 30718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260806 23:59:52.433892 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 30718
I20260806 23:59:52.434026 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260806 23:59:52.434309 9009 external_mini_cluster.cc:1428] Setting key c26ce1f96c0f24934481fa4feeecf96f
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:52.443631 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.431167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:44669 (local address 127.8.204.126:37545)
0806 23:59:52.431310 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.431314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.432576 (+ 1262us) server_negotiation.cc:407] Connection header received
0806 23:59:52.433461 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.433465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.433525 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.433605 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:52.434980 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.435519 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.436627 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.436860 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.436973 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.439747 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:52.439754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:52.439760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:52.439794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:52.441532 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.442015 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.442020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.442023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.442128 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.442484 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.442489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.442491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.442687 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:52.442851 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.443329 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.443452 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:59:52.444371 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.431525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:44669)
0806 23:59:52.432433 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.432474 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.433258 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.433786 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.433792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.434225 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:52.434818 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.434842 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.435650 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.435653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.436498 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.436507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.437013 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.437017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.437180 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.437866 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:52.437893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:52.439575 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:52.441648 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.441653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.441663 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.441907 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.442234 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.442236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.442237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.442372 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.442846 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:52.442852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:52.443088 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.443285 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.443586 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":558,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260806 23:59:52.445605 30899 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260806 23:59:52.445860 30899 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:52.446393 30899 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260806 23:59:52.447559 30443 ts_manager.cc:194] Registered new tserver with Master: c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260806 23:59:52.448241 30443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:44669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260806 23:59:52.461493 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:59:52.572899 30905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:59:52.573179 30905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:59:52.573230 30905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:59:52.577678 30905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:59:52.577730 30905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:59:52.577790 30905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260806 23:59:52.583279 30905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:59:52.584223 30905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:59:52.585223 30905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:59:52.595468 30913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:52.595541 30905 server_base.cc:1034] running on GCE node
W20260806 23:59:52.595481 30916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:59:52.595517 30912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:59:52.596148 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:59:52.596797 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:59:52.597982 30905 hybrid_clock.cc:648] HybridClock initialized: now 1786060792597970 us; error 27 us; skew 500 ppm
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:59:52.600826 30905 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260806 23:59:52.601727 30905 webserver.cc:460] Webserver started at http://127.8.204.67:43551/ using document root <none> and password file <none>
I20260806 23:59:52.602239 30905 fs_manager.cc:359] Metadata directory not provided
I20260806 23:59:52.602298 30905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:59:52.602460 30905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:59:52.604635 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "ad8702c7cdc545019843815b209d160e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "4d1af86409e74dde1769e3d062bec20a"
server_key_iv: "91acd9a568440f0749c5d593fe92588c"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.605119 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "ad8702c7cdc545019843815b209d160e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "4d1af86409e74dde1769e3d062bec20a"
server_key_iv: "91acd9a568440f0749c5d593fe92588c"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.608990 30905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:59:52.611213 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.612385 30905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:59:52.612498 30905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "ad8702c7cdc545019843815b209d160e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "4d1af86409e74dde1769e3d062bec20a"
server_key_iv: "91acd9a568440f0749c5d593fe92588c"
server_key_version: "encryptionkey@0"
I20260806 23:59:52.612588 30905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:59:52.640861 30905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:59:52.643801 30905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:59:52.643965 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:59:52.644550 30905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:59:52.645399 30905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:59:52.645452 30905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.645503 30905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:59:52.645529 30905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:59:52.655387 30905 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:35289
I20260806 23:59:52.655411 31088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:35289 every 8 connection(s)
I20260806 23:59:52.656324 30905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260806 23:59:52.657788 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 30905
I20260806 23:59:52.657914 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260806 23:59:52.658290 9009 external_mini_cluster.cc:1428] Setting key 6730d24e23cd67f43d43c9fa4894e820
Aug 06 23:59:52 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260806 23:59:52.669968 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.658214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:34987 (local address 127.8.204.126:37545)
0806 23:59:52.658375 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.658380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.659668 (+ 1288us) server_negotiation.cc:407] Connection header received
0806 23:59:52.660770 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.660776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.660835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.660902 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:52.662046 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.662537 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.663823 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.664028 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.664099 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.666543 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:52.666546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:52.666548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:52.666570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:52.668002 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.668504 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.668508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.668509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.668589 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.668922 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.668926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.668928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.669100 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:52.669205 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.669676 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.669802 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 23:59:52.670781 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.658610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:34987)
0806 23:59:52.659469 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.659542 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.660539 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.661082 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.661089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.661441 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:59:52.661911 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.661924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.662643 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.662646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.663676 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.663686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.664154 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.664157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.664306 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.664852 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:59:52.664870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:59:52.666364 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:59:52.668123 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.668128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.668141 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.668398 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.668682 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:59:52.668685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:59:52.668687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:59:52.668824 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:59:52.669227 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:59:52.669232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:59:52.669451 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.669629 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.669922 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":137,"threads_started":1}
I20260806 23:59:52.671939 31089 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260806 23:59:52.672194 31089 heartbeater.cc:464] Registering TS with master...
I20260806 23:59:52.672750 31089 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260806 23:59:52.673839 30440 ts_manager.cc:194] Registered new tserver with Master: ad8702c7cdc545019843815b209d160e (127.8.204.67:35289)
I20260806 23:59:52.674467 30440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:34987
I20260806 23:59:52.684221 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:59:52.692729 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.686143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.8.204.126:37545)
0806 23:59:52.686256 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.686260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.686466 (+ 206us) server_negotiation.cc:407] Connection header received
0806 23:59:52.686558 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.686560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.686595 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.686658 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:59:52.687304 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.687770 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.688509 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.688702 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.688772 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.689957 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:59:52.689960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:59:52.689962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:59:52.689982 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:59:52.691397 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.691754 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.691756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.691757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.691825 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:59:52.692064 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:59:52.692066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:59:52.692067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:59:52.692250 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:59:52.692350 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.692533 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.692617 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:59:52.695530 30440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:59:52.698275 30440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:59:52.712837 31108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.706779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:35289 (local address 127.0.0.1:49100)
0806 23:59:52.707699 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.707714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.707807 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.708684 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.708686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.708700 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:52.708820 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.708829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.710224 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.710227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.711457 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.711465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712280 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.712401 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712440 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.712573 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.712689 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":111,"thread_start_us":88,"threads_started":1}
I20260806 23:59:52.712846 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.707444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.8.204.67:35289)
0806 23:59:52.708310 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.708321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.708341 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:59:52.708401 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.708405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.708557 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.708694 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:52.708951 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.710074 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.711566 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712152 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712280 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712428 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.712606 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.712709 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":153,"threads_started":1}
I20260806 23:59:52.713094 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.706613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:37889 (local address 127.0.0.1:55918)
0806 23:59:52.707242 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.707278 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.707737 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.708564 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.708571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.708596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:52.708782 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.708791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.709911 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.709914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.711275 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.711286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712617 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.712739 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712781 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.712856 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.712952 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":88,"thread_start_us":105,"threads_started":1}
I20260806 23:59:52.713244 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.707610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.8.204.65:33789)
0806 23:59:52.707992 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.708007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.708061 (+ 54us) server_negotiation.cc:407] Connection header received
0806 23:59:52.708265 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.708270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.708443 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.708579 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:52.709107 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.710174 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.711558 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712381 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712473 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712600 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.712967 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.713061 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":98,"threads_started":1}
I20260806 23:59:52.713536 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.707388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:33789 (local address 127.0.0.1:51330)
0806 23:59:52.707989 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.708002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.708085 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.708720 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.708724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.708744 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:52.708969 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.708978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.710282 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.710286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.711423 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.711434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712651 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.712757 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712797 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.712857 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.713406 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":646,"thread_start_us":77,"threads_started":1}
I20260806 23:59:52.713555 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.706770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.8.204.66:37889)
0806 23:59:52.707613 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.707626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.707651 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:59:52.707946 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.707952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.708184 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.708338 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:52.708921 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.709776 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.711408 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.712281 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.712395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.712546 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.713297 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.713404 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":115,"threads_started":1}
I20260806 23:59:52.716270 30801 tablet_service.cc:1467] Processing CreateTablet for tablet 6722ed6d364846eaa7c95547d10b0825 (DEFAULT_TABLE table=test-table [id=d2aae5f1ee274f31ac0457bcd49713dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:52.716259 31000 tablet_service.cc:1467] Processing CreateTablet for tablet 6722ed6d364846eaa7c95547d10b0825 (DEFAULT_TABLE table=test-table [id=d2aae5f1ee274f31ac0457bcd49713dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:52.716495 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 6722ed6d364846eaa7c95547d10b0825 (DEFAULT_TABLE table=test-table [id=d2aae5f1ee274f31ac0457bcd49713dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:59:52.717338 30801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6722ed6d364846eaa7c95547d10b0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:52.717417 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6722ed6d364846eaa7c95547d10b0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:52.717424 31000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6722ed6d364846eaa7c95547d10b0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:59:52.723194 31112 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Bootstrap starting.
I20260806 23:59:52.724704 31114 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Bootstrap starting.
I20260806 23:59:52.725906 31112 tablet_bootstrap.cc:654] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:52.726449 31114 tablet_bootstrap.cc:654] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:52.726677 31112 log.cc:824] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:52.727074 31113 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Bootstrap starting.
I20260806 23:59:52.727672 31114 log.cc:824] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:52.728703 31112 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: No bootstrap required, opened a new log
I20260806 23:59:52.728919 31112 ts_tablet_manager.cc:1397] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:59:52.729027 31113 tablet_bootstrap.cc:654] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Neither blocks nor log segments found. Creating new log.
I20260806 23:59:52.729741 31114 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: No bootstrap required, opened a new log
I20260806 23:59:52.729882 31113 log.cc:824] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:59:52.729925 31114 ts_tablet_manager.cc:1397] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:59:52.731755 31113 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: No bootstrap required, opened a new log
I20260806 23:59:52.731928 31113 ts_tablet_manager.cc:1397] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:59:52.732157 31112 raft_consensus.cc:348] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.732405 31112 raft_consensus.cc:374] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:52.732463 31112 raft_consensus.cc:729] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61b77292d074595971a8f922a4691d6, State: Initialized, Role: FOLLOWER
I20260806 23:59:52.732944 31112 consensus_queue.cc:260] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.733790 31112 ts_tablet_manager.cc:1428] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:59:52.733716 31114 raft_consensus.cc:348] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.733969 31114 raft_consensus.cc:374] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:52.734030 31114 raft_consensus.cc:729] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8702c7cdc545019843815b209d160e, State: Initialized, Role: FOLLOWER
I20260806 23:59:52.734201 30899 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
I20260806 23:59:52.734449 31114 consensus_queue.cc:260] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.735107 31089 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
I20260806 23:59:52.736289 31114 ts_tablet_manager.cc:1428] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 23:59:52.737353 31113 raft_consensus.cc:348] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.737601 31113 raft_consensus.cc:374] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:59:52.737665 31113 raft_consensus.cc:729] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9671421c00422ba185d0947ab6f97e, State: Initialized, Role: FOLLOWER
I20260806 23:59:52.738075 31113 consensus_queue.cc:260] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.738770 30713 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
I20260806 23:59:52.739060 31113 ts_tablet_manager.cc:1428] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260806 23:59:52.845409 31118 raft_consensus.cc:482] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:59:52.845678 31118 raft_consensus.cc:504] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.846968 31118 leader_election.cc:288] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789)
I20260806 23:59:52.851687 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.847344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:42251 (local address 127.8.204.67:35289)
0806 23:59:52.847486 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.847491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.847507 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:59:52.847565 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.847568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.847647 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.847739 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:52.848094 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.848880 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.850403 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.851016 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.851137 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.851199 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.851491 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.851557 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:59:52.851699 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.847551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:53241 (local address 127.8.204.65:33789)
0806 23:59:52.847726 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.847731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.847772 (+ 41us) server_negotiation.cc:407] Connection header received
0806 23:59:52.847934 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.847936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.847979 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.848074 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:59:52.848317 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.849030 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.850397 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.851020 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.851137 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.851189 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.851491 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.851557 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:59:52.851722 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.847200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:35289 (local address 127.8.204.66:42251)
0806 23:59:52.847321 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.847339 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.847446 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.847776 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.847779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.847821 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:52.847970 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.847976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.848991 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.848993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.850265 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.850277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.851144 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.851148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.851335 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.851382 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.851462 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.851568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":77,"spinlock_wait_cycles":27264}
I20260806 23:59:52.851732 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.847265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:33789 (local address 127.8.204.66:53241)
0806 23:59:52.847681 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.847692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:59:52.847828 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:59:52.848085 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:59:52.848091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:59:52.848112 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:59:52.848210 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.848214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.849138 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.849141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.850265 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:59:52.850277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.851144 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.851148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:59:52.851321 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.851377 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 23:59:52.851462 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 23:59:52.851568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":59,"spinlock_wait_cycles":6528,"thread_start_us":164,"threads_started":1}
I20260806 23:59:52.852478 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6722ed6d364846eaa7c95547d10b0825" candidate_uuid: "c61b77292d074595971a8f922a4691d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8702c7cdc545019843815b209d160e" is_pre_election: true
I20260806 23:59:52.852486 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6722ed6d364846eaa7c95547d10b0825" candidate_uuid: "c61b77292d074595971a8f922a4691d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9671421c00422ba185d0947ab6f97e" is_pre_election: true
I20260806 23:59:52.852780 30646 raft_consensus.cc:2454] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61b77292d074595971a8f922a4691d6 in term 0.
I20260806 23:59:52.852790 31026 raft_consensus.cc:2454] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c61b77292d074595971a8f922a4691d6 in term 0.
I20260806 23:59:52.853284 30739 leader_election.cc:302] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e, c61b77292d074595971a8f922a4691d6; no voters:
I20260806 23:59:52.853567 31118 raft_consensus.cc:2781] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:59:52.853667 31118 raft_consensus.cc:482] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:59:52.853719 31118 raft_consensus.cc:3037] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:52.854796 31118 raft_consensus.cc:504] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.855181 31118 leader_election.cc:288] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [CANDIDATE]: Term 1 election: Requested vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789)
I20260806 23:59:52.855546 31026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6722ed6d364846eaa7c95547d10b0825" candidate_uuid: "c61b77292d074595971a8f922a4691d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8702c7cdc545019843815b209d160e"
I20260806 23:59:52.855567 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6722ed6d364846eaa7c95547d10b0825" candidate_uuid: "c61b77292d074595971a8f922a4691d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9671421c00422ba185d0947ab6f97e"
I20260806 23:59:52.855670 31026 raft_consensus.cc:3037] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:52.855684 30646 raft_consensus.cc:3037] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:59:52.856712 30646 raft_consensus.cc:2454] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61b77292d074595971a8f922a4691d6 in term 1.
I20260806 23:59:52.856750 31026 raft_consensus.cc:2454] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c61b77292d074595971a8f922a4691d6 in term 1.
I20260806 23:59:52.857048 30739 leader_election.cc:302] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e, c61b77292d074595971a8f922a4691d6; no voters:
I20260806 23:59:52.857237 31118 raft_consensus.cc:2781] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:59:52.857479 31118 raft_consensus.cc:686] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 1 LEADER]: Becoming Leader. State: Replica: c61b77292d074595971a8f922a4691d6, State: Running, Role: LEADER
I20260806 23:59:52.857811 31118 consensus_queue.cc:237] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260806 23:59:52.861258 30443 catalog_manager.cc:5495] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c61b77292d074595971a8f922a4691d6 (127.8.204.66). New cstate: current_term: 1 leader_uuid: "c61b77292d074595971a8f922a4691d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } health_report { overall_health: UNKNOWN } } }
I20260806 23:59:52.903911 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.900255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.8.204.66:37889)
0806 23:59:52.900396 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:59:52.900403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:59:52.900419 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:59:52.900610 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:59:52.900613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:59:52.900668 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:59:52.900779 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:59:52.901026 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.901619 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.902455 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:59:52.902674 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:59:52.902809 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:59:52.903140 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:59:52.903504 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:59:52.903646 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 23:59:52.903723 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 23:59:52.903777 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 23:59:52.907333 31090 tablet.cc:2369] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:59:52.909725 31026 raft_consensus.cc:1261] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Refusing update from remote peer c61b77292d074595971a8f922a4691d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:52.909723 30646 raft_consensus.cc:1261] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Refusing update from remote peer c61b77292d074595971a8f922a4691d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:59:52.910406 31118 consensus_queue.cc:1035] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:52.910880 31125 consensus_queue.cc:1035] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:59:52.918334 31133 mvcc.cc:205] Tried to move back new op lower bound from 7315705007751417856 to 7315705007548121088. Current Snapshot: MvccSnapshot[applied={T|T < 7315705007751417856}]
I20260806 23:59:52.918691 31132 mvcc.cc:205] Tried to move back new op lower bound from 7315705007751417856 to 7315705007548121088. Current Snapshot: MvccSnapshot[applied={T|T < 7315705007751417856}]
I20260806 23:59:52.921111 31134 mvcc.cc:205] Tried to move back new op lower bound from 7315705007751417856 to 7315705007548121088. Current Snapshot: MvccSnapshot[applied={T|T < 7315705007751417856}]
I20260806 23:59:52.926801 30443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:52.926940 30443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:59:52.929183 30443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Sending DeleteTablet for 3 replicas of tablet 6722ed6d364846eaa7c95547d10b0825
I20260806 23:59:52.929982 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 30528
I20260806 23:59:52.930003 30619 tablet_service.cc:1514] Processing DeleteTablet for tablet 6722ed6d364846eaa7c95547d10b0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:51330
I20260806 23:59:52.930037 31000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6722ed6d364846eaa7c95547d10b0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:49100
I20260806 23:59:52.930109 30801 tablet_service.cc:1514] Processing DeleteTablet for tablet 6722ed6d364846eaa7c95547d10b0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:55918
I20260806 23:59:52.939067 31141 tablet_replica.cc:330] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: stopping tablet replica
I20260806 23:59:52.939446 31141 raft_consensus.cc:2229] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:59:52.939807 31141 raft_consensus.cc:2258] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:52.940186 31143 tablet_replica.cc:330] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: stopping tablet replica
I20260806 23:59:52.940515 31143 raft_consensus.cc:2229] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:59:52.940917 31143 raft_consensus.cc:2258] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:59:52.941705 31141 ts_tablet_manager.cc:1905] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:59:52.942265 31143 ts_tablet_manager.cc:1905] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 23:59:52.944079 30415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:33789 (error 108)
I20260806 23:59:52.944679 31141 ts_tablet_manager.cc:1918] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:59:52.944849 31141 log.cc:1196] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/6722ed6d364846eaa7c95547d10b0825
I20260806 23:59:52.945007 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 30718
I20260806 23:59:52.949564 31141 ts_tablet_manager.cc:1939] T 6722ed6d364846eaa7c95547d10b0825 P ad8702c7cdc545019843815b209d160e: Deleting consensus metadata
I20260806 23:59:52.950798 30419 catalog_manager.cc:4841] TS ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): tablet 6722ed6d364846eaa7c95547d10b0825 (table test-table [id=d2aae5f1ee274f31ac0457bcd49713dc]) successfully deleted
I20260806 23:59:52.952265 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.951963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:33789 (local address 127.0.0.1:51338)
0806 23:59:52.952123 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.952190 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 23:59:52.952430 30415 catalog_manager.cc:4584] TS ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6722ed6d364846eaa7c95547d10b0825: Network error: Client connection negotiation failed: client connection to 127.8.204.65:33789: connect: Connection refused (error 111)
I20260806 23:59:52.957506 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 30905
I20260806 23:59:52.957701 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:59:52.957405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:37889 (local address 127.0.0.1:55924)
0806 23:59:52.957532 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 23:59:52.957600 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.66:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 23:59:52.967896 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 30383
I20260807 00:00:12.977032 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--webserver_interface=127.8.204.126
--webserver_port=34543
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:13.096056 31146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:13.096388 31146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:13.096462 31146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:13.100515 31146 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 00:00:13.100612 31146 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 00:00:13.100641 31146 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 00:00:13.100670 31146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:13.100713 31146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:13.100757 31146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:13.100811 31146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:13.100852 31146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:13.106493 31146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=34543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:13.107769 31146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:13.108875 31146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:13.123929 31158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.123955 31154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.123955 31155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:13.124121 31146 server_base.cc:1034] running on GCE node
I20260807 00:00:13.125017 31146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:13.126287 31146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:13.127467 31146 hybrid_clock.cc:648] HybridClock initialized: now 1786060813127436 us; error 48 us; skew 500 ppm
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:13.130524 31146 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:00:13.131619 31146 webserver.cc:460] Webserver started at http://127.8.204.126:34543/ using document root <none> and password file <none>
I20260807 00:00:13.132263 31146 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:13.132377 31146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:13.136471 31146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260807 00:00:13.138816 31167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.140017 31146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:13.140178 31146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "cdb1b82a60a4420f88872857c7f77c82"
format_stamp: "Formatted at 2026-08-06 23:59:51 on dist-test-slave-39l8"
server_key: "836fccdd2cd64e3e03cc8354cee22adb"
server_key_iv: "4834fa25aea63ea7761458aac149d2b9"
server_key_version: "encryptionkey@0"
I20260807 00:00:13.140807 31146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:13.162177 31146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:13.165670 31146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:13.165884 31146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:13.173517 31146 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37545
I20260807 00:00:13.173828 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37545 every 8 connection(s)
I20260807 00:00:13.174675 31146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:13.175539 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31146
I20260807 00:00:13.175959 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:33789
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=34227
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:00:13.179507 31246 sys_catalog.cc:263] Verifying existing consensus state
I20260807 00:00:13.182181 31246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap starting.
I20260807 00:00:13.207147 31246 log.cc:824] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:13.218297 31246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 00:00:13.218775 31246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap complete.
I20260807 00:00:13.224604 31246 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:13.225134 31246 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Initialized, Role: FOLLOWER
I20260807 00:00:13.225698 31246 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:13.225852 31246 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:13.225930 31246 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:13.226039 31246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 1 FOLLOWER]: Advancing to term 2
I20260807 00:00:13.228605 31246 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:13.228974 31246 leader_election.cc:302] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb1b82a60a4420f88872857c7f77c82; no voters:
I20260807 00:00:13.229329 31246 leader_election.cc:288] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 00:00:13.229460 31250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 00:00:13.229717 31250 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 LEADER]: Becoming Leader. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Running, Role: LEADER
I20260807 00:00:13.230063 31250 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:13.230628 31246 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:13.231685 31254 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260807 00:00:13.231644 31256 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb1b82a60a4420f88872857c7f77c82. Latest consensus state: current_term: 2 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260807 00:00:13.231782 31254 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:13.231782 31256 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:13.235864 31272 catalog_manager.cc:1259] Loaded cluster ID: a452088a89fb49ad85ce9f07435bb8bd
I20260807 00:00:13.235927 31272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: loading cluster ID for follower catalog manager: success
I20260807 00:00:13.238407 31272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: acquiring CA information for follower catalog manager: success
I20260807 00:00:13.239981 31272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 00:00:13.240130 31265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:13.242144 31265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:13.242575 31265 catalog_manager.cc:1259] Loaded cluster ID: a452088a89fb49ad85ce9f07435bb8bd
I20260807 00:00:13.242657 31265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:13.243312 31265 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:13.243707 31265 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Loaded TSK: 0
I20260807 00:00:13.259613 31265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Generated new TSK 1
I20260807 00:00:13.259863 31265 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 00:00:13.322899 31248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:13.323153 31248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:13.323228 31248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:13.327198 31248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:13.327271 31248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:13.327355 31248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:13.332754 31248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:33789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=34227
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:13.333727 31248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:13.334741 31248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:13.345186 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.345340 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.345186 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:13.345820 31248 server_base.cc:1034] running on GCE node
I20260807 00:00:13.346226 31248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:13.346913 31248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:13.348100 31248 hybrid_clock.cc:648] HybridClock initialized: now 1786060813348063 us; error 36 us; skew 500 ppm
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:13.351018 31248 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260807 00:00:13.352156 31248 webserver.cc:460] Webserver started at http://127.8.204.65:34227/ using document root <none> and password file <none>
I20260807 00:00:13.352873 31248 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:13.352982 31248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:13.357950 31248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:00:13.360591 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.362205 31248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:00:13.362337 31248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "ac9671421c00422ba185d0947ab6f97e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "81f965793886b36a36c188235ad8b5ab"
server_key_iv: "41bd90bf2927c66ae3e68313844369da"
server_key_version: "encryptionkey@0"
I20260807 00:00:13.363065 31248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:13.406412 31248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:13.410037 31248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:13.410241 31248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:13.411021 31248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:13.412209 31305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 00:00:13.416435 31248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 00:00:13.416510 31248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260807 00:00:13.416599 31248 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 00:00:13.418252 31248 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 00:00:13.418298 31248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 00:00:13.418574 31305 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Bootstrap starting.
I20260807 00:00:13.429509 31248 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:33789
I20260807 00:00:13.429572 31468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:33789 every 8 connection(s)
I20260807 00:00:13.430634 31248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:13.434974 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31248
I20260807 00:00:13.435513 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:37889
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=37163
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:00:13.438454 31305 log.cc:824] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Log is configured to *not* fsync() on all Append() calls
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:13.446857 31305 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 00:00:13.447266 31305 tablet_bootstrap.cc:492] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Bootstrap complete.
I20260807 00:00:13.448683 31305 ts_tablet_manager.cc:1397] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.007s
I20260807 00:00:13.451431 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.433055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:35333)
0807 00:00:13.433687 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.433746 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.435093 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.436488 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.436497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.436982 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:13.437612 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.437633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.439631 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.439635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.440915 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.440927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.442059 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.442063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.442244 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.443030 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:13.443051 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:13.445010 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:13.447893 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.447902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.447918 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.448291 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.448745 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.448751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.448755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.448928 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.449621 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:13.449627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:13.449906 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.450108 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.450433 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":128,"threads_started":1}
I20260807 00:00:13.451465 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.433052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35333 (local address 127.8.204.126:37545)
0807 00:00:13.433500 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.433520 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.433884 (+ 364us) server_negotiation.cc:407] Connection header received
0807 00:00:13.435595 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.435613 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.436010 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.436378 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.437817 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.439447 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.441060 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.441895 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.442115 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.445228 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.445237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.445255 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.445302 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.447762 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.448428 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.448438 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.448447 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.448622 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.449045 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.449049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.449051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.449489 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.449629 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.450250 (+ 621us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.450548 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":153,"threads_started":1}
I20260807 00:00:13.451596 31305 raft_consensus.cc:348] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.452155 31305 raft_consensus.cc:729] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9671421c00422ba185d0947ab6f97e, State: Initialized, Role: FOLLOWER
I20260807 00:00:13.452638 31305 consensus_queue.cc:260] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.453558 31305 ts_tablet_manager.cc:1428] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260807 00:00:13.454346 31469 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260807 00:00:13.454600 31469 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:13.455111 31469 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260807 00:00:13.457620 31191 ts_manager.cc:194] Registered new tserver with Master: ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789)
I20260807 00:00:13.461683 31191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:35333
I20260807 00:00:13.464010 31469 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.65@KRBTEST.COM
I20260807 00:00:13.472458 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.460840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:33789 (local address 127.0.0.1:50622)
0807 00:00:13.461507 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.461542 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.461701 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.462233 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.462239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.462667 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:13.462849 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.462858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.464249 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.464252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.465010 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.465020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.465720 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.465723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.465800 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.466437 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:13.466452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:13.468110 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:13.470586 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.470595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.470600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.470908 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.471228 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.471231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.471233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.471322 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.471782 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:13.471788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:13.471913 (+ 125us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.472057 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.472273 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":109,"threads_started":1}
I20260807 00:00:13.472462 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.460883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.8.204.65:33789)
0807 00:00:13.461270 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.461281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.461638 (+ 357us) server_negotiation.cc:407] Connection header received
0807 00:00:13.461924 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.461928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.462087 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.462231 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.463000 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.464125 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.465139 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.465518 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.465618 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.468291 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.468302 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.468318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.468352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.470433 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.471038 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.471044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.471046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.471133 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.471421 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.471424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.471426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.471662 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.471809 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.472162 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.472321 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":110,"threads_started":1}
I20260807 00:00:13.473479 31375 tablet_service.cc:1514] Processing DeleteTablet for tablet 6722ed6d364846eaa7c95547d10b0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:59:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:50622
I20260807 00:00:13.474479 31480 tablet_replica.cc:330] stopping tablet replica
I20260807 00:00:13.474852 31480 raft_consensus.cc:2229] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:13.475103 31480 raft_consensus.cc:2258] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:13.476045 31480 ts_tablet_manager.cc:1905] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:13.479554 31480 ts_tablet_manager.cc:1918] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:13.479676 31480 log.cc:1196] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal/wals/6722ed6d364846eaa7c95547d10b0825
I20260807 00:00:13.479943 31480 ts_tablet_manager.cc:1939] T 6722ed6d364846eaa7c95547d10b0825 P ac9671421c00422ba185d0947ab6f97e: Deleting consensus metadata
I20260807 00:00:13.481539 31170 catalog_manager.cc:4841] TS ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789): tablet 6722ed6d364846eaa7c95547d10b0825 (table test-table [id=d2aae5f1ee274f31ac0457bcd49713dc]) successfully deleted
W20260807 00:00:13.561343 31475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:13.561618 31475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:13.561664 31475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:13.565780 31475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:13.565845 31475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:13.565936 31475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:13.572611 31475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:37889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=37163
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:13.573805 31475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:13.574934 31475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:13.585201 31488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.585186 31491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.585178 31494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:13.585664 31475 server_base.cc:1034] running on GCE node
I20260807 00:00:13.586205 31475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:13.586966 31475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:13.588195 31475 hybrid_clock.cc:648] HybridClock initialized: now 1786060813588160 us; error 49 us; skew 500 ppm
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:13.591317 31475 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260807 00:00:13.592269 31475 webserver.cc:460] Webserver started at http://127.8.204.66:37163/ using document root <none> and password file <none>
I20260807 00:00:13.592861 31475 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:13.593024 31475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:13.597209 31475 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 00:00:13.599674 31506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.600968 31475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 00:00:13.601107 31475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "c61b77292d074595971a8f922a4691d6"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "e846cbd346250eb96eabd065c4c6d345"
server_key_iv: "57f0ea2f7c9f1388537e63c1bbec6eb5"
server_key_version: "encryptionkey@0"
I20260807 00:00:13.601675 31475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:13.628213 31475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:13.631722 31475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:13.631944 31475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:13.632625 31475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:13.633734 31514 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260807 00:00:13.637486 31514 ts_tablet_manager.cc:1852] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 00:00:13.637877 31514 ts_tablet_manager.cc:1905] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:13.641714 31514 ts_tablet_manager.cc:1918] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:13.641813 31514 log.cc:1196] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/6722ed6d364846eaa7c95547d10b0825
I20260807 00:00:13.642200 31514 ts_tablet_manager.cc:1939] T 6722ed6d364846eaa7c95547d10b0825 P c61b77292d074595971a8f922a4691d6: Deleting consensus metadata
I20260807 00:00:13.642514 31475 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 00:00:13.642618 31475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260807 00:00:13.642743 31475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:13.642786 31475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.652035 31475 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:37889
I20260807 00:00:13.652079 31656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:37889 every 8 connection(s)
I20260807 00:00:13.652878 31475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:13.653968 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31475
I20260807 00:00:13.654439 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:35289
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=43551
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:13.666640 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.654857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:46053 (local address 127.8.204.126:37545)
0807 00:00:13.655029 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.655036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.655912 (+ 876us) server_negotiation.cc:407] Connection header received
0807 00:00:13.656859 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.656862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.656928 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.657025 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.658178 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.658783 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.659859 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.660094 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.660203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.662630 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.662637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.662644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.662680 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.664441 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.664939 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.664945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.664950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.665054 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.665380 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.665386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.665389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.665648 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.665790 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.666272 (+ 482us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.666427 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:13.667364 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.655254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:46053)
0807 00:00:13.655753 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.655804 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.656652 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.657207 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.657213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.657556 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:13.658033 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.658047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.658914 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.658918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.659735 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.659745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.660244 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.660248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.660405 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.661016 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:13.661032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:13.662457 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:13.664575 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.664580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.664591 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.664838 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.665153 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.665155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.665156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.665283 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.665775 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:13.665780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:13.666010 (+ 230us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.666183 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.666421 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":143,"threads_started":1}
I20260807 00:00:13.668592 31657 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260807 00:00:13.668846 31657 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:13.669430 31657 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260807 00:00:13.670502 31191 ts_manager.cc:194] Registered new tserver with Master: c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
I20260807 00:00:13.671113 31191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:46053
W20260807 00:00:13.769731 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:13.769987 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:13.770062 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:13.774238 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:13.774297 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:13.774397 31664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:13.779523 31664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:35289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=43551
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:13.780534 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:13.781579 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:13.790874 31670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:13.791008 31664 server_base.cc:1034] running on GCE node
W20260807 00:00:13.790933 31675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:13.790904 31672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:13.791512 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:13.792061 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:13.793218 31664 hybrid_clock.cc:648] HybridClock initialized: now 1786060813793203 us; error 25 us; skew 500 ppm
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:13.795934 31664 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260807 00:00:13.796782 31664 webserver.cc:460] Webserver started at http://127.8.204.67:43551/ using document root <none> and password file <none>
I20260807 00:00:13.797324 31664 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:13.797416 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:13.801139 31664 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260807 00:00:13.803169 31685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.804332 31664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:13.804462 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "ad8702c7cdc545019843815b209d160e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "4d1af86409e74dde1769e3d062bec20a"
server_key_iv: "91acd9a568440f0749c5d593fe92588c"
server_key_version: "encryptionkey@0"
I20260807 00:00:13.804955 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:13.827903 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:13.830781 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:13.830929 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:13.831507 31664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:13.832398 31664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:13.832454 31664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.832530 31664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:13.832566 31664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:13.841913 31664 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:35289
I20260807 00:00:13.841951 31848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:35289 every 8 connection(s)
I20260807 00:00:13.842905 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
Aug 07 00:00:13 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:13.851986 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31664
I20260807 00:00:13.855923 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.844825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:57961 (local address 127.8.204.126:37545)
0807 00:00:13.844978 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.844983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.846031 (+ 1048us) server_negotiation.cc:407] Connection header received
0807 00:00:13.846923 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.846926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.846974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.847053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.848114 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.848626 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.849691 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.849898 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.849974 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.852274 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.852278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.852281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.852309 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.853881 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.854384 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.854388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.854390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.854470 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.854845 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.854848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.854850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.855033 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.855135 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.855630 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.855751 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:00:13.856837 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.845238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:57961)
0807 00:00:13.845888 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.845929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.846684 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.847186 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.847192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.847533 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:13.847992 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.848005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.848730 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.848733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.849589 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.849598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.850019 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.850022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.850159 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.850662 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:13.850677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:13.852100 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:13.854001 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.854007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.854022 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.854279 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.854562 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:13.854565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:13.854567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:13.854759 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:13.855146 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:13.855153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:13.855393 (+ 240us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.855568 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.855881 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":116,"threads_started":1}
I20260807 00:00:13.858162 31849 heartbeater.cc:347] Connected to a master server at 127.8.204.126:37545
I20260807 00:00:13.858407 31849 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:13.858948 31849 heartbeater.cc:511] Master 127.8.204.126:37545 requested a full tablet report, sending...
I20260807 00:00:13.860154 31191 ts_manager.cc:194] Registered new tserver with Master: ad8702c7cdc545019843815b209d160e (127.8.204.67:35289)
I20260807 00:00:13.860888 31191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:57961
I20260807 00:00:13.861057 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.852883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.8.204.126:37545)
0807 00:00:13.853147 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.853152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.853165 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:00:13.853279 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.853282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.853326 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.853413 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.854145 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.854655 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.855537 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.855763 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.855865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.857413 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.857417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.857419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.857443 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.859076 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.859760 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.859763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.859765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.859855 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.860161 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.860167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.860172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.860363 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.860515 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.860744 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.860845 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260807 00:00:13.863039 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:00:13.866457 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.863633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.8.204.126:37545)
0807 00:00:13.863739 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.863742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.863919 (+ 177us) server_negotiation.cc:407] Connection header received
0807 00:00:13.864020 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.864022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.864114 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.864194 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:13.864378 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.864822 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.865593 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.865791 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.865863 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.866100 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:13.866207 (+ 107us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 00:00:13.866380 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56270: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 00:00:13.866530 31855 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56270: token expired
W20260807 00:00:13.866889 31094 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.204.126:37545)
I20260807 00:00:13.873479 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.867244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.8.204.126:37545)
0807 00:00:13.867344 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.867348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.867362 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:00:13.867445 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.867448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.867482 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.867545 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:13.868131 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.868634 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.869375 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.869559 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.869643 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.870862 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:13.870865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:13.870867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:13.870888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:13.872128 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.872484 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.872486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.872487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.872556 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:13.872792 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:13.872810 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:13.872815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:13.873002 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:13.873104 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.873295 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.873380 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 00:00:13.875828 31191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:00:13.878931 31191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:00:13.886440 31375 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6235c7cfea40089b2027002525185c (DEFAULT_TABLE table=test-table [id=bdf12e027fb048c2be6663715985a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:13.887190 31375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6235c7cfea40089b2027002525185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:13.890020 31305 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Bootstrap starting.
I20260807 00:00:13.890908 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.885263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:35289 (local address 127.0.0.1:53596)
0807 00:00:13.885600 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.885618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.885698 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.886553 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.886556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.886576 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:13.886677 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.886683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.888224 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.888228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.889306 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.889317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.890378 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.890382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.890488 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.890529 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.890587 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.890739 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260807 00:00:13.890969 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.885500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:37889 (local address 127.0.0.1:40938)
0807 00:00:13.885991 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:13.886004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:13.886102 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:13.886776 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:13.886779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:13.886798 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:13.886914 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.886920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.888004 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.888007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.889246 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:13.889257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.890412 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.890416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:13.890533 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.890558 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 00:00:13.890652 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 00:00:13.890761 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260807 00:00:13.890980 31305 tablet_bootstrap.cc:654] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:13.891285 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.885726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.8.204.66:37889)
0807 00:00:13.886108 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.886120 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.886166 (+ 46us) server_negotiation.cc:407] Connection header received
0807 00:00:13.886233 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.886237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.886375 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.886545 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:13.887051 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.887884 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.889399 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.890304 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.890399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.890553 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.890719 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.890823 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":105,"threads_started":1}
I20260807 00:00:13.892338 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:13.885466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.8.204.67:35289)
0807 00:00:13.886013 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:13.886027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:13.886052 (+ 25us) server_negotiation.cc:407] Connection header received
0807 00:00:13.886135 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:13.886144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:13.886368 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:13.886545 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:13.886841 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.888094 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.889434 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:13.890230 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:13.891192 (+ 962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:13.891334 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:13.892077 (+ 743us) server_negotiation.cc:299] Negotiation successful
0807 00:00:13.892178 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":163,"threads_started":1}
I20260807 00:00:13.892607 31305 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: No bootstrap required, opened a new log
I20260807 00:00:13.892688 31305 ts_tablet_manager.cc:1397] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 00:00:13.893113 31305 raft_consensus.cc:348] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.892745 31575 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6235c7cfea40089b2027002525185c (DEFAULT_TABLE table=test-table [id=bdf12e027fb048c2be6663715985a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:13.893235 31305 raft_consensus.cc:374] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:13.893277 31305 raft_consensus.cc:729] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9671421c00422ba185d0947ab6f97e, State: Initialized, Role: FOLLOWER
I20260807 00:00:13.893400 31305 consensus_queue.cc:260] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.893759 31305 ts_tablet_manager.cc:1428] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 00:00:13.893859 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6235c7cfea40089b2027002525185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:13.895049 31748 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6235c7cfea40089b2027002525185c (DEFAULT_TABLE table=test-table [id=bdf12e027fb048c2be6663715985a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:13.896077 31748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6235c7cfea40089b2027002525185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:13.898695 31514 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Bootstrap starting.
I20260807 00:00:13.900961 31514 tablet_bootstrap.cc:654] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:13.901857 31514 log.cc:824] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:13.903568 31868 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Bootstrap starting.
I20260807 00:00:13.903877 31514 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: No bootstrap required, opened a new log
I20260807 00:00:13.904057 31514 ts_tablet_manager.cc:1397] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:13.905876 31868 tablet_bootstrap.cc:654] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:13.906665 31868 log.cc:824] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:13.907147 31514 raft_consensus.cc:348] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.907442 31514 raft_consensus.cc:374] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:13.907506 31514 raft_consensus.cc:729] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61b77292d074595971a8f922a4691d6, State: Initialized, Role: FOLLOWER
I20260807 00:00:13.907981 31514 consensus_queue.cc:260] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.908663 31868 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: No bootstrap required, opened a new log
I20260807 00:00:13.908681 31657 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
I20260807 00:00:13.908854 31868 ts_tablet_manager.cc:1397] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 00:00:13.908985 31514 ts_tablet_manager.cc:1428] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:00:13.912122 31868 raft_consensus.cc:348] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.912328 31868 raft_consensus.cc:374] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:13.912364 31868 raft_consensus.cc:729] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8702c7cdc545019843815b209d160e, State: Initialized, Role: FOLLOWER
I20260807 00:00:13.912672 31868 consensus_queue.cc:260] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:13.913280 31849 heartbeater.cc:503] Master 127.8.204.126:37545 was elected leader, sending a full tablet report...
I20260807 00:00:13.913482 31868 ts_tablet_manager.cc:1428] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260807 00:00:13.932021 31470 tablet.cc:2369] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:14.039063 31477 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:14.039318 31477 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:14.040572 31477 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
I20260807 00:00:14.044586 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.040804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:35289 (local address 127.8.204.65:51561)
0807 00:00:14.041269 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:14.041286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:14.041409 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:14.041686 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:14.041689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:14.041705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:14.041824 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:14.041831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.042568 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.042571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:14.043420 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:14.043430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.044086 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.044089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:14.044197 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:14.044244 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 00:00:14.044334 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 00:00:14.044410 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
I20260807 00:00:14.044603 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.040942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41491 (local address 127.8.204.66:37889)
0807 00:00:14.041088 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:14.041094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:14.041223 (+ 129us) server_negotiation.cc:407] Connection header received
0807 00:00:14.041376 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:14.041381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:14.041438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:14.041518 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:14.041824 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.042369 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.043577 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.043977 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.044112 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:14.044166 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:14.044384 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 00:00:14.044471 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 00:00:14.044862 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.040946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:51561 (local address 127.8.204.67:35289)
0807 00:00:14.041100 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:14.041105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:14.041386 (+ 281us) server_negotiation.cc:407] Connection header received
0807 00:00:14.041521 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:14.041524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:14.041577 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:14.041657 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:14.041942 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.042456 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.043547 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.043956 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.044076 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:14.044140 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:14.044355 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 00:00:14.044429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:14.044649 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.040804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:37889 (local address 127.8.204.65:41491)
0807 00:00:14.041129 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:14.041143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:14.041252 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:14.041542 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:14.041545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:14.041562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:14.041703 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:14.041710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.042487 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.042493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:14.043425 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:14.043435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.044118 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.044122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:14.044216 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:14.044282 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 00:00:14.044361 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 00:00:14.044494 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":49,"thread_start_us":76,"threads_started":1}
I20260807 00:00:14.045639 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6235c7cfea40089b2027002525185c" candidate_uuid: "ac9671421c00422ba185d0947ab6f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61b77292d074595971a8f922a4691d6" is_pre_election: true
I20260807 00:00:14.045640 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6235c7cfea40089b2027002525185c" candidate_uuid: "ac9671421c00422ba185d0947ab6f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8702c7cdc545019843815b209d160e" is_pre_election: true
I20260807 00:00:14.045977 31779 raft_consensus.cc:2454] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9671421c00422ba185d0947ab6f97e in term 0.
I20260807 00:00:14.045977 31600 raft_consensus.cc:2454] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9671421c00422ba185d0947ab6f97e in term 0.
I20260807 00:00:14.046517 31298 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e, ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6; no voters:
I20260807 00:00:14.046818 31477 raft_consensus.cc:2781] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:00:14.046912 31477 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:00:14.046974 31477 raft_consensus.cc:3037] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:14.048003 31477 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:14.048358 31477 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 1 election: Requested vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
I20260807 00:00:14.048707 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6235c7cfea40089b2027002525185c" candidate_uuid: "ac9671421c00422ba185d0947ab6f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8702c7cdc545019843815b209d160e"
I20260807 00:00:14.048712 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6235c7cfea40089b2027002525185c" candidate_uuid: "ac9671421c00422ba185d0947ab6f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61b77292d074595971a8f922a4691d6"
I20260807 00:00:14.048836 31779 raft_consensus.cc:3037] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:14.048836 31600 raft_consensus.cc:3037] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:14.050110 31600 raft_consensus.cc:2454] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9671421c00422ba185d0947ab6f97e in term 1.
I20260807 00:00:14.050110 31779 raft_consensus.cc:2454] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9671421c00422ba185d0947ab6f97e in term 1.
I20260807 00:00:14.050448 31299 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e, ad8702c7cdc545019843815b209d160e; no voters:
I20260807 00:00:14.050662 31477 raft_consensus.cc:2781] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:14.050877 31477 raft_consensus.cc:686] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 LEADER]: Becoming Leader. State: Replica: ac9671421c00422ba185d0947ab6f97e, State: Running, Role: LEADER
I20260807 00:00:14.051146 31477 consensus_queue.cc:237] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:14.054054 31191 catalog_manager.cc:5495] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9671421c00422ba185d0947ab6f97e (127.8.204.65). New cstate: current_term: 1 leader_uuid: "ac9671421c00422ba185d0947ab6f97e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } health_report { overall_health: HEALTHY } } }
I20260807 00:00:14.085527 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.081889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.8.204.65:33789)
0807 00:00:14.082228 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:14.082235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:14.082248 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:00:14.082327 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:14.082329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:14.082384 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:14.082494 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:14.082759 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.083271 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.084052 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:14.084348 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:14.084426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:14.084640 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:14.085087 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:00:14.085171 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:14.085256 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 00:00:14.085314 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260807 00:00:14.089607 31779 raft_consensus.cc:1261] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Refusing update from remote peer ac9671421c00422ba185d0947ab6f97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:14.089605 31600 raft_consensus.cc:1261] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Refusing update from remote peer ac9671421c00422ba185d0947ab6f97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:14.090307 31877 consensus_queue.cc:1035] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:14.090507 31477 consensus_queue.cc:1035] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 00:00:14.093760 31850 tablet.cc:2369] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:14.098629 31884 mvcc.cc:205] Tried to move back new op lower bound from 7315705094504239104 to 7315705094355972096. Current Snapshot: MvccSnapshot[applied={T|T < 7315705094504239104}]
I20260807 00:00:14.107187 31191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:14.107388 31191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:14.109977 31191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Sending DeleteTablet for 3 replicas of tablet 8b6235c7cfea40089b2027002525185c
I20260807 00:00:14.110845 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 31248
I20260807 00:00:14.110780 31748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6235c7cfea40089b2027002525185c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:14 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:53596
I20260807 00:00:14.110827 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6235c7cfea40089b2027002525185c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:14 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:40938
I20260807 00:00:14.115715 31895 tablet_replica.cc:330] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: stopping tablet replica
I20260807 00:00:14.116094 31895 raft_consensus.cc:2229] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:14.116425 31895 raft_consensus.cc:2258] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:14.118665 31896 tablet_replica.cc:330] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: stopping tablet replica
I20260807 00:00:14.119028 31896 raft_consensus.cc:2229] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:14.119052 31895 ts_tablet_manager.cc:1905] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:14.119371 31896 raft_consensus.cc:2258] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:14.120860 31896 ts_tablet_manager.cc:1905] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:14.121927 31895 ts_tablet_manager.cc:1918] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:14.122017 31895 log.cc:1196] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/8b6235c7cfea40089b2027002525185c
I20260807 00:00:14.122344 31895 ts_tablet_manager.cc:1939] T 8b6235c7cfea40089b2027002525185c P c61b77292d074595971a8f922a4691d6: Deleting consensus metadata
I20260807 00:00:14.123636 31896 ts_tablet_manager.cc:1918] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:14.123764 31896 log.cc:1196] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/8b6235c7cfea40089b2027002525185c
I20260807 00:00:14.123796 31170 catalog_manager.cc:4841] TS c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): tablet 8b6235c7cfea40089b2027002525185c (table test-table [id=bdf12e027fb048c2be6663715985a0e6]) successfully deleted
I20260807 00:00:14.124122 31896 ts_tablet_manager.cc:1939] T 8b6235c7cfea40089b2027002525185c P ad8702c7cdc545019843815b209d160e: Deleting consensus metadata
I20260807 00:00:14.125157 31172 catalog_manager.cc:4841] TS ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): tablet 8b6235c7cfea40089b2027002525185c (table test-table [id=bdf12e027fb048c2be6663715985a0e6]) successfully deleted
W20260807 00:00:14.125279 31170 connection.cc:535] client connection to 127.8.204.65:33789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 00:00:14.125448 31170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 00:00:14.126293 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 31475
I20260807 00:00:14.126555 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:14.126244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:33789 (local address 127.0.0.1:50642)
0807 00:00:14.126382 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:14.126458 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 00:00:14.131712 31170 catalog_manager.cc:4584] TS ac9671421c00422ba185d0947ab6f97e (127.8.204.65:33789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b6235c7cfea40089b2027002525185c: Network error: Client connection negotiation failed: client connection to 127.8.204.65:33789: connect: Connection refused (error 111)
I20260807 00:00:14.143558 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 31664
I20260807 00:00:14.155566 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 31146
I20260807 00:00:34.167184 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--webserver_interface=127.8.204.126
--webserver_port=34543
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:34.287235 31900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:34.287508 31900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:34.287549 31900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:34.291474 31900 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 00:00:34.291524 31900 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 00:00:34.291539 31900 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 00:00:34.291553 31900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:34.291571 31900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:34.291589 31900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:34.291605 31900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:34.291620 31900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:34.297006 31900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=34543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:34.298060 31900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:34.299062 31900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 00:00:34.308058 31900 server_base.cc:1034] running on GCE node
W20260807 00:00:34.308015 31907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.308015 31911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.308034 31908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:34.308903 31900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:34.310027 31900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:34.311192 31900 hybrid_clock.cc:648] HybridClock initialized: now 1786060834311169 us; error 31 us; skew 500 ppm
Aug 07 00:00:34 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:34.318979 31900 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:00:34.319983 31900 webserver.cc:460] Webserver started at http://127.8.204.126:34543/ using document root <none> and password file <none>
I20260807 00:00:34.320577 31900 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:34.320641 31900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:34.324790 31900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 00:00:34.327080 31919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.328346 31900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 00:00:34.328454 31900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "cdb1b82a60a4420f88872857c7f77c82"
format_stamp: "Formatted at 2026-08-06 23:59:51 on dist-test-slave-39l8"
server_key: "836fccdd2cd64e3e03cc8354cee22adb"
server_key_iv: "4834fa25aea63ea7761458aac149d2b9"
server_key_version: "encryptionkey@0"
I20260807 00:00:34.328982 31900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:34.339234 31900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:34.342137 31900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:34.342309 31900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:34.349653 31900 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37545
I20260807 00:00:34.349956 31993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37545 every 8 connection(s)
I20260807 00:00:34.350802 31900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:34.355326 31994 sys_catalog.cc:263] Verifying existing consensus state
I20260807 00:00:34.356207 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31900
I20260807 00:00:34.356673 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:33789
--local_ip_for_outbound_sockets=127.8.204.65
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=34227
--webserver_interface=127.8.204.65
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:00:34.357200 31994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap starting.
I20260807 00:00:34.383253 31994 log.cc:824] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:34.398737 31994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 00:00:34.399262 31994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Bootstrap complete.
I20260807 00:00:34.405326 31994 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:34.405881 31994 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Initialized, Role: FOLLOWER
I20260807 00:00:34.406476 31994 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:34.406639 31994 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:34.406733 31994 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:34.406853 31994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 2 FOLLOWER]: Advancing to term 3
I20260807 00:00:34.409541 31994 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:34.409920 31994 leader_election.cc:302] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb1b82a60a4420f88872857c7f77c82; no voters:
I20260807 00:00:34.410295 31994 leader_election.cc:288] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 00:00:34.410447 32000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 00:00:34.410799 32000 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [term 3 LEADER]: Becoming Leader. State: Replica: cdb1b82a60a4420f88872857c7f77c82, State: Running, Role: LEADER
I20260807 00:00:34.411172 32000 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } }
I20260807 00:00:34.411680 31994 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:34.412700 32004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb1b82a60a4420f88872857c7f77c82. Latest consensus state: current_term: 3 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260807 00:00:34.412828 32004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:34.413189 32001 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cdb1b82a60a4420f88872857c7f77c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb1b82a60a4420f88872857c7f77c82" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37545 } } }
I20260807 00:00:34.413275 32001 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:34.413872 32013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:34.419638 32013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:34.420385 32013 catalog_manager.cc:1259] Loaded cluster ID: a452088a89fb49ad85ce9f07435bb8bd
I20260807 00:00:34.420459 32013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:34.423158 32013 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:34.423985 32013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Loaded TSK: 0
I20260807 00:00:34.424079 32013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb1b82a60a4420f88872857c7f77c82: Loaded TSK: 1
F20260807 00:00:34.425348 32013 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3a2783a9ad google::LogMessage::Fail() at ??:0
@ 0x7f3a2783d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3a2783a4de google::LogMessage::Flush() at ??:0
@ 0x7f3a2783ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3a2a4fe18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3a2a4f6f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3a321a37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3a3217d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3a3217e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3a321bfdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3a321de82b std::function<>::operator()() at ??:0
@ 0x7f3a3217db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3a3217ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3a3217b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3a321bf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3a29a8c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3a29a8cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3a29a8e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3a29a7d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3a2b0c16db start_thread at ??:0
@ 0x7f3a2633861f clone at ??:0
W20260807 00:00:34.528219 31996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:34.528470 31996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:34.528515 31996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:34.532634 31996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:34.532691 31996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:34.532778 31996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:34.538057 31996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:33789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=34227
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:34.539247 31996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:34.540287 31996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:34.550797 32058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.551071 32056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.551075 32054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:34.552089 31996 server_base.cc:1034] running on GCE node
I20260807 00:00:34.552560 31996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:34.553376 31996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:34.554591 31996 hybrid_clock.cc:648] HybridClock initialized: now 1786060834554588 us; error 47 us; skew 500 ppm
Aug 07 00:00:34 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:34.558010 31996 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260807 00:00:34.559170 31996 webserver.cc:460] Webserver started at http://127.8.204.65:34227/ using document root <none> and password file <none>
I20260807 00:00:34.559875 31996 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:34.559970 31996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:34.565047 31996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260807 00:00:34.567731 32066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.569239 31996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:00:34.569362 31996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "ac9671421c00422ba185d0947ab6f97e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "81f965793886b36a36c188235ad8b5ab"
server_key_iv: "41bd90bf2927c66ae3e68313844369da"
server_key_version: "encryptionkey@0"
I20260807 00:00:34.570057 31996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:34.586392 31996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:34.590055 31996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:34.590257 31996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:34.590963 31996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:34.592299 32076 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 00:00:34.596536 31996 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 00:00:34.596612 31996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260807 00:00:34.596674 31996 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 00:00:34.598945 31996 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 00:00:34.599018 31996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:34.599341 32076 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Bootstrap starting.
I20260807 00:00:34.610321 31996 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:33789
I20260807 00:00:34.610348 32222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:33789 every 8 connection(s)
I20260807 00:00:34.611630 31996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:34.616220 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 31996
I20260807 00:00:34.616755 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:37889
--local_ip_for_outbound_sockets=127.8.204.66
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=37163
--webserver_interface=127.8.204.66
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:00:34.622043 32076 log.cc:824] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:34.632251 32076 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 00:00:34.632725 32076 tablet_bootstrap.cc:492] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Bootstrap complete.
I20260807 00:00:34.634311 32076 ts_tablet_manager.cc:1397] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.004s
I20260807 00:00:34.637390 32076 raft_consensus.cc:348] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:34.637913 32076 raft_consensus.cc:729] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9671421c00422ba185d0947ab6f97e, State: Initialized, Role: FOLLOWER
I20260807 00:00:34.638417 32076 consensus_queue.cc:260] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:34.639526 32076 ts_tablet_manager.cc:1428] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
W20260807 00:00:34.664592 32226 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:00:34.614072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:56199)
0807 00:00:34.614694 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.614819 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:34.616083 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:34.663649 (+ 47566us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":153,"threads_started":1}
W20260807 00:00:34.665210 32224 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:37545 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:37545: Client connection negotiation failed: client connection to 127.8.204.126:37545: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 00:00:34.665967 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:34.665687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:45111)
0807 00:00:34.665828 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.665902 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 00:00:34.666746 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:34.666500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:33265)
0807 00:00:34.666616 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.666667 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 00:00:34.666941 32224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 00:00:34.741416 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:34.741677 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:34.741720 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:34.745728 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:34.745791 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:34.745849 32227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:34.751376 32227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:37889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=37163
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:34.752435 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:34.753458 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:34.763304 32244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.763316 32241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.763303 32240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:34.763396 32227 server_base.cc:1034] running on GCE node
I20260807 00:00:34.763994 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:34.764607 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:34.765802 32227 hybrid_clock.cc:648] HybridClock initialized: now 1786060834765794 us; error 32 us; skew 500 ppm
Aug 07 00:00:34 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:34.768546 32227 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260807 00:00:34.769449 32227 webserver.cc:460] Webserver started at http://127.8.204.66:37163/ using document root <none> and password file <none>
I20260807 00:00:34.770018 32227 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:34.770112 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:34.773885 32227 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:34.776127 32252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.777369 32227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:00:34.777542 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "c61b77292d074595971a8f922a4691d6"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "e846cbd346250eb96eabd065c4c6d345"
server_key_iv: "57f0ea2f7c9f1388537e63c1bbec6eb5"
server_key_version: "encryptionkey@0"
I20260807 00:00:34.778079 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:34.790467 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:34.793582 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:34.793795 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:34.794370 32227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:34.795269 32227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:34.795343 32227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.795418 32227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:34.795462 32227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.804656 32227 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:37889
I20260807 00:00:34.804689 32412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:37889 every 8 connection(s)
I20260807 00:00:34.805612 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:34.809142 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:34.807656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:55511)
0807 00:00:34.808339 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.808442 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":121,"threads_started":1}
I20260807 00:00:34.809571 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 32227
W20260807 00:00:34.809710 32413 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:37545 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:37545: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
I20260807 00:00:34.810092 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:35289
--local_ip_for_outbound_sockets=127.8.204.67
--tserver_master_addrs=127.8.204.126:37545
--webserver_port=43551
--webserver_interface=127.8.204.67
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:00:34.810381 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:34.810100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:43667)
0807 00:00:34.810252 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.810314 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 00:00:34.811136 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:34.810925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:33933)
0807 00:00:34.811047 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:34.811090 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 00:00:34.811326 32413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 00:00:34.926488 32416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:34.926893 32416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:34.926949 32416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:34.930962 32416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:34.931022 32416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:34.931082 32416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:34.936390 32416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:35289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=43551
--tserver_master_addrs=127.8.204.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:34.937583 32416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:34.938596 32416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:34.948546 32425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:34.948577 32422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:34.948652 32416 server_base.cc:1034] running on GCE node
W20260807 00:00:34.948554 32423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:34.949226 32416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:34.949918 32416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:34.951107 32416 hybrid_clock.cc:648] HybridClock initialized: now 1786060834951069 us; error 49 us; skew 500 ppm
Aug 07 00:00:34 dist-test-slave-39l8 krb5kdc[30366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:34.953996 32416 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260807 00:00:34.954923 32416 webserver.cc:460] Webserver started at http://127.8.204.67:43551/ using document root <none> and password file <none>
I20260807 00:00:34.955467 32416 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:34.955528 32416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:34.959383 32416 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 00:00:34.961524 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.962872 32416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:34.963020 32416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "ad8702c7cdc545019843815b209d160e"
format_stamp: "Formatted at 2026-08-06 23:59:52 on dist-test-slave-39l8"
server_key: "4d1af86409e74dde1769e3d062bec20a"
server_key_iv: "91acd9a568440f0749c5d593fe92588c"
server_key_version: "encryptionkey@0"
I20260807 00:00:34.963613 32416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:34.986320 32416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:34.990000 32416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:34.990227 32416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:34.990912 32416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:34.991838 32416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:34.991888 32416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:34.991952 32416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:34.992007 32416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:35.001564 32416 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:35289
I20260807 00:00:35.001642 32582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:35289 every 8 connection(s)
I20260807 00:00:35.002722 32416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:00:35.006641 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:35.005015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:45597)
0807 00:00:35.005762 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:35.005881 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":112,"threads_started":1}
W20260807 00:00:35.007287 32584 heartbeater.cc:644] Failed to heartbeat to 127.8.204.126:37545 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.126:37545: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
I20260807 00:00:35.007970 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:35.007714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:36309)
0807 00:00:35.007855 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:35.007910 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 00:00:35.008315 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 32416
I20260807 00:00:35.008917 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:35.008691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:54339)
0807 00:00:35.008820 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:35.008866 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 00:00:35.009102 32584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 00:00:35.667927 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:35.667458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:37515)
0807 00:00:35.667717 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:35.667772 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260807 00:00:35.812282 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:35.811832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:46553)
0807 00:00:35.812123 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:35.812180 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":68,"threads_started":1}
I20260807 00:00:36.010026 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.009616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:34993)
0807 00:00:36.009879 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:36.009931 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260807 00:00:36.036463 32595 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:36.036705 32595 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:36.037786 32595 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
Aug 07 00:00:36 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.67@KRBTEST.COM
Aug 07 00:00:36 dist-test-slave-39l8 krb5kdc[30366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.66@KRBTEST.COM
I20260807 00:00:36.051440 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.037995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:37889 (local address 127.8.204.65:44539)
0807 00:00:36.038400 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:36.038418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:36.038511 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:36.040089 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:36.040096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:36.040842 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:36.041131 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:36.041145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.042410 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.042413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:36.043414 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:36.043424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.044142 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.044146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:36.044374 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:36.045080 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:36.045093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:36.046987 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:36.049045 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:36.049051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:36.049296 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:36.049649 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:36.049654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:36.049797 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:36.050400 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:36.050406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:36.050667 (+ 261us) client_negotiation.cc:769] Sending connection context
0807 00:00:36.050903 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 00:00:36.051252 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":179,"spinlock_wait_cycles":9088,"thread_start_us":71,"threads_started":1}
I20260807 00:00:36.051455 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.038196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:44539 (local address 127.8.204.66:37889)
0807 00:00:36.038620 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:36.038642 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:36.038677 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:00:36.039393 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:36.039399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:36.039733 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:36.040039 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:36.041306 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.042250 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.043537 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.044016 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.044194 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:36.047178 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:36.047182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:36.047194 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:36.047221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:36.048923 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049412 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:36.049420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:36.049430 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:36.049540 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049896 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:36.049900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:36.049901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:36.050269 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:36.050425 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:36.051013 (+ 588us) server_negotiation.cc:299] Negotiation successful
0807 00:00:36.051266 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":84,"threads_started":1}
I20260807 00:00:36.052003 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.038196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:39899 (local address 127.8.204.67:35289)
0807 00:00:36.038606 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:36.038630 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:36.038668 (+ 38us) server_negotiation.cc:407] Connection header received
0807 00:00:36.039414 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:36.039420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:36.039731 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:36.040041 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:36.041312 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.042255 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.043540 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.044018 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.044200 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:36.046932 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:36.046937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:36.046949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:36.046975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:36.048840 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049405 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:36.049410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:36.049414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:36.049523 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049891 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:36.049895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:36.049897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:36.050269 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:36.050425 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:36.051279 (+ 854us) server_negotiation.cc:299] Negotiation successful
0807 00:00:36.051805 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":123,"threads_started":1}
I20260807 00:00:36.053129 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.037995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:35289 (local address 127.8.204.65:39899)
0807 00:00:36.038191 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:36.038207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:36.038317 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:36.040096 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:36.040099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:36.040588 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:36.041132 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:36.041145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.042397 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.042400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:36.043415 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:36.043426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:36.044154 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:36.044157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:36.044375 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:36.045030 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:36.045047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:36.046660 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:36.048973 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:36.048979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:36.048992 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:36.049275 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:36.049639 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:36.049642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:36.049643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:36.049792 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:36.050402 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:36.050407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:36.050669 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 00:00:36.050903 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 00:00:36.053003 (+ 2100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":545,"spinlock_wait_cycles":8832}
W20260807 00:00:36.054087 32070 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
W20260807 00:00:36.054986 32071 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
I20260807 00:00:36.055104 32071 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e; no voters: ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6
I20260807 00:00:36.055392 32595 raft_consensus.cc:2726] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 00:00:36.669301 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.668823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:36813)
0807 00:00:36.669143 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:36.669204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":78,"threads_started":1}
I20260807 00:00:36.813354 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:36.812976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:41481)
0807 00:00:36.813223 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:36.813277 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260807 00:00:37.011333 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:37.010926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:46225)
0807 00:00:37.011197 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:37.011248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
I20260807 00:00:37.670504 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:37.670093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:58243)
0807 00:00:37.670354 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:37.670408 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260807 00:00:37.814517 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:37.814134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:49633)
0807 00:00:37.814378 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:37.814431 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260807 00:00:37.905707 32613 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:37.905846 32613 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:37.906373 32613 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
W20260807 00:00:37.907284 32071 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
W20260807 00:00:37.907395 32070 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
I20260807 00:00:37.907459 32070 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e; no voters: ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6
I20260807 00:00:37.907613 32613 raft_consensus.cc:2726] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 00:00:38.012773 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:38.012192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:40557)
0807 00:00:38.012536 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:38.012624 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260807 00:00:38.671774 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:38.671344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:40519)
0807 00:00:38.671612 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:38.671668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260807 00:00:38.815768 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:38.815345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:60489)
0807 00:00:38.815625 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:38.815682 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
I20260807 00:00:39.013994 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:39.013553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:41783)
0807 00:00:39.013837 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:39.013895 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260807 00:00:39.673120 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:39.672686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:36331)
0807 00:00:39.672969 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:39.673023 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260807 00:00:39.817246 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:39.816833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:57613)
0807 00:00:39.817089 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:39.817143 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260807 00:00:40.015589 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:40.015092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:45697)
0807 00:00:40.015425 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:40.015489 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":142,"threads_started":1}
I20260807 00:00:40.159054 32628 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:40.159209 32628 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:40.159633 32628 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
W20260807 00:00:40.160472 32070 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
W20260807 00:00:40.160581 32071 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
I20260807 00:00:40.160647 32071 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e; no voters: ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6
I20260807 00:00:40.160780 32628 raft_consensus.cc:2726] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 00:00:40.674538 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:40.674058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:54679)
0807 00:00:40.674370 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:40.674430 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260807 00:00:40.818576 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:40.818130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:39723)
0807 00:00:40.818438 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:40.818489 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260807 00:00:41.016716 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:41.016309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:50339)
0807 00:00:41.016564 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:41.016616 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260807 00:00:41.675868 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:41.675402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:41315)
0807 00:00:41.675714 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:41.675769 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260807 00:00:41.819955 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:41.819439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:33151)
0807 00:00:41.819795 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:41.819863 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":74,"threads_started":1}
I20260807 00:00:42.018486 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:42.017714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:49849)
0807 00:00:42.018277 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:42.018363 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":142,"threads_started":1}
I20260807 00:00:42.372186 32640 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:42.372327 32640 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:42.372882 32640 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
W20260807 00:00:42.373755 32070 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
W20260807 00:00:42.373901 32071 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
I20260807 00:00:42.374014 32071 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e; no voters: ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6
I20260807 00:00:42.374228 32640 raft_consensus.cc:2726] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 00:00:42.677245 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:42.676754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:34655)
0807 00:00:42.677070 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:42.677142 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260807 00:00:42.821324 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:42.820864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:58465)
0807 00:00:42.821164 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:42.821229 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260807 00:00:43.020008 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:43.019533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:57627)
0807 00:00:43.019838 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:43.019907 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260807 00:00:43.678664 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:43.678198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:46039)
0807 00:00:43.678495 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:43.678562 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260807 00:00:43.822456 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:43.822056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:47727)
0807 00:00:43.822309 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:43.822365 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260807 00:00:44.021225 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:44.020787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:43195)
0807 00:00:44.021063 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:44.021126 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260807 00:00:44.680035 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:44.679591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:46019)
0807 00:00:44.679878 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:44.679936 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260807 00:00:44.823660 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:44.823237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:52793)
0807 00:00:44.823506 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:44.823562 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260807 00:00:45.022452 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:45.022006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:41999)
0807 00:00:45.022288 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:45.022346 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260807 00:00:45.681373 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:45.680892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:60387)
0807 00:00:45.681195 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:45.681261 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260807 00:00:45.825057 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:45.824533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:59963)
0807 00:00:45.824874 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:45.824945 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260807 00:00:46.023765 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:46.023284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:57979)
0807 00:00:46.023597 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:46.023660 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260807 00:00:46.682600 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:46.682141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:55605)
0807 00:00:46.682403 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:46.682457 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260807 00:00:46.826504 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:46.826028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:42897)
0807 00:00:46.826346 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:46.826405 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260807 00:00:47.025090 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:47.024649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:54757)
0807 00:00:47.024931 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:47.024990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260807 00:00:47.683761 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:47.683329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:40399)
0807 00:00:47.683607 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:47.683662 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260807 00:00:47.827755 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:47.827297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:45263)
0807 00:00:47.827597 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:47.827656 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260807 00:00:48.026360 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:48.025940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:55523)
0807 00:00:48.026199 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:48.026259 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260807 00:00:48.251042 32665 raft_consensus.cc:482] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:48.251184 32665 raft_consensus.cc:504] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8702c7cdc545019843815b209d160e" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 35289 } } peers { permanent_uuid: "c61b77292d074595971a8f922a4691d6" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 37889 } } peers { permanent_uuid: "ac9671421c00422ba185d0947ab6f97e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 33789 } }
I20260807 00:00:48.251689 32665 leader_election.cc:288] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad8702c7cdc545019843815b209d160e (127.8.204.67:35289), c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889)
W20260807 00:00:48.252498 32070 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c61b77292d074595971a8f922a4691d6 (127.8.204.66:37889): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
W20260807 00:00:48.252595 32071 leader_election.cc:341] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad8702c7cdc545019843815b209d160e (127.8.204.67:35289): Not found: Tablet not found: 8b6235c7cfea40089b2027002525185c
I20260807 00:00:48.252724 32071 leader_election.cc:302] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9671421c00422ba185d0947ab6f97e; no voters: ad8702c7cdc545019843815b209d160e, c61b77292d074595971a8f922a4691d6
I20260807 00:00:48.252910 32665 raft_consensus.cc:2726] T 8b6235c7cfea40089b2027002525185c P ac9671421c00422ba185d0947ab6f97e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 00:00:48.684901 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:48.684459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:33807)
0807 00:00:48.684746 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:48.684801 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
I20260807 00:00:48.829105 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:48.828655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:57377)
0807 00:00:48.828943 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:48.829008 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260807 00:00:49.027617 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:49.027191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:39657)
0807 00:00:49.027472 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:49.027524 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260807 00:00:49.686331 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:49.685838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.65:50653)
0807 00:00:49.686143 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:49.686208 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260807 00:00:49.830471 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:49.830014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.66:44029)
0807 00:00:49.830292 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:49.830376 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
I20260807 00:00:50.013993 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 31996
I20260807 00:00:50.028093 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 32227
I20260807 00:00:50.034461 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:50.032618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37545 (local address 127.8.204.67:41705)
0807 00:00:50.034235 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:50.034311 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1502,"thread_start_us":521,"threads_started":1}
I20260807 00:00:50.042639 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 32416
2026-08-07T00:00:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59667 ms)
[----------] 30 tests from SecurityITest (240804 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:00:50 dist-test-slave-39l8 krb5kdc[32675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:00:50 dist-test-slave-39l8 krb5kdc[32675](info): set up 2 sockets
Aug 07 00:00:50 dist-test-slave-39l8 krb5kdc[32675](info): commencing operation
krb5kdc: starting...
W20260807 00:00:51.253640 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:00:51.374874 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:43099
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:51.485590 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:51.485829 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:51.485867 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:51.489780 32692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:51.489833 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:51.489853 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:51.489869 32692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:51.489884 32692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:51.494984 32692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:43099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:51.495878 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:51.496836 32692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:51.505384 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:51.505528 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:51.505600 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:51.505676 32692 server_base.cc:1034] running on GCE node
I20260807 00:00:51.506316 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:51.507417 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:51.508606 32692 hybrid_clock.cc:648] HybridClock initialized: now 1786060851508588 us; error 29 us; skew 500 ppm
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:51.511555 32692 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:00:51.512392 32692 webserver.cc:460] Webserver started at http://127.8.204.126:33607/ using document root <none> and password file <none>
I20260807 00:00:51.512936 32692 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:51.513028 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:51.513267 32692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:51.515352 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "70183f60b1c24c51b2c5886c28046bf3"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "0bd0f3aab092777e50c490af4db41008"
server_key_iv: "b58563799e2f197aa182711a9b816f28"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.515902 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "70183f60b1c24c51b2c5886c28046bf3"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "0bd0f3aab092777e50c490af4db41008"
server_key_iv: "b58563799e2f197aa182711a9b816f28"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.519496 32692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 00:00:51.521801 32709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:51.523044 32692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:51.523190 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "70183f60b1c24c51b2c5886c28046bf3"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "0bd0f3aab092777e50c490af4db41008"
server_key_iv: "b58563799e2f197aa182711a9b816f28"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.523298 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:51.536228 32692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:51.539021 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:51.539218 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:51.545934 32692 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:43099
I20260807 00:00:51.546006 314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:43099 every 8 connection(s)
I20260807 00:00:51.546880 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:51.549995 315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:51.551169 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 32692
I20260807 00:00:51.551326 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:00:51.551712 9009 external_mini_cluster.cc:1428] Setting key 21fad9809ab85d547aeeba85679e3a22
I20260807 00:00:51.557636 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: Bootstrap starting.
I20260807 00:00:51.560653 315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:51.561646 315 log.cc:824] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: Log is configured to *not* fsync() on all Append() calls
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:51.563680 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: No bootstrap required, opened a new log
I20260807 00:00:51.567158 315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } }
I20260807 00:00:51.567396 315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:51.567476 315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70183f60b1c24c51b2c5886c28046bf3, State: Initialized, Role: FOLLOWER
I20260807 00:00:51.567751 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:51.553262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.8.204.126:43099)
0807 00:00:51.553742 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:51.553761 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:51.553805 (+ 44us) server_negotiation.cc:407] Connection header received
0807 00:00:51.554629 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:51.554652 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:51.555079 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:51.555432 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:51.556443 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.558279 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.559537 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.560213 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.560333 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:51.563259 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:51.563264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:51.563276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:51.563307 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:51.564996 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:51.565435 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:51.565441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:51.565452 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:51.565555 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:51.565850 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:51.565853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:51.565855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:51.566225 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:51.566436 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:51.566690 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 00:00:51.566985 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260807 00:00:51.567968 315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } }
I20260807 00:00:51.568142 315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:51.568210 315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:51.568310 315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:51.569396 315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } }
I20260807 00:00:51.569805 315 leader_election.cc:302] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70183f60b1c24c51b2c5886c28046bf3; no voters:
I20260807 00:00:51.570197 315 leader_election.cc:288] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:00:51.570288 325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:51.570525 325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [term 1 LEADER]: Becoming Leader. State: Replica: 70183f60b1c24c51b2c5886c28046bf3, State: Running, Role: LEADER
I20260807 00:00:51.570883 325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } }
I20260807 00:00:51.571439 315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:51.572613 327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70183f60b1c24c51b2c5886c28046bf3. Latest consensus state: current_term: 1 leader_uuid: "70183f60b1c24c51b2c5886c28046bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } } }
I20260807 00:00:51.572754 327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:51.573134 334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:51.573074 326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70183f60b1c24c51b2c5886c28046bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70183f60b1c24c51b2c5886c28046bf3" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 43099 } } }
I20260807 00:00:51.573773 326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:51.576083 334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:51.581534 334 catalog_manager.cc:1347] Generated new cluster ID: 5c3f3c2f5218470dabff77539c7e1981
I20260807 00:00:51.581594 334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:51.587682 334 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:00:51.588838 334 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:51.596138 334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: Generated new TSK 0
I20260807 00:00:51.596853 334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260807 00:00:51.627962 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:51.747740 352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:51.748032 352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:51.748078 352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:51.752238 352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:51.752315 352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:51.752378 352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:51.757898 352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:51.759127 352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:51.760169 352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:51.770403 359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:51.770545 357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:51.770403 361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:51.770931 352 server_base.cc:1034] running on GCE node
I20260807 00:00:51.771381 352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:51.772296 352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:51.773480 352 hybrid_clock.cc:648] HybridClock initialized: now 1786060851773463 us; error 43 us; skew 500 ppm
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:51.776432 352 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260807 00:00:51.777354 352 webserver.cc:460] Webserver started at http://127.8.204.65:34697/ using document root <none> and password file <none>
I20260807 00:00:51.777959 352 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:51.778054 352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:51.778306 352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:51.780495 352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "cec3517ee27e4131b4006af4bae107c4"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "811156de11237277f6b17add2b13493b"
server_key_iv: "6c2203b5610ad1db4d248c104475d60a"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.781092 352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "cec3517ee27e4131b4006af4bae107c4"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "811156de11237277f6b17add2b13493b"
server_key_iv: "6c2203b5610ad1db4d248c104475d60a"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.785105 352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 00:00:51.787756 373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:51.789079 352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:00:51.789239 352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "cec3517ee27e4131b4006af4bae107c4"
format_stamp: "Formatted at 2026-08-07 00:00:51 on dist-test-slave-39l8"
server_key: "811156de11237277f6b17add2b13493b"
server_key_iv: "6c2203b5610ad1db4d248c104475d60a"
server_key_version: "encryptionkey@0"
I20260807 00:00:51.789363 352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:51.800838 352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:51.804069 352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:51.804298 352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:51.804927 352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:51.805804 352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:51.805878 352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:51.805954 352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:51.806005 352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:51.815572 352 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:39351
I20260807 00:00:51.815632 532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:39351 every 8 connection(s)
I20260807 00:00:51.816553 352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:51.825140 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 352
I20260807 00:00:51.825289 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:00:51.825635 9009 external_mini_cluster.cc:1428] Setting key ab3b7cf43b09585ddc9b50f701396311
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:51.831257 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:51.818470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:60279 (local address 127.8.204.126:43099)
0807 00:00:51.818656 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:51.818662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:51.819567 (+ 905us) server_negotiation.cc:407] Connection header received
0807 00:00:51.820905 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:51.820908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:51.820958 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:51.821040 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:51.822187 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.822776 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.823893 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.824122 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.824201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:51.827161 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:51.827165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:51.827168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:51.827195 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:51.829015 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:51.829575 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:51.829578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:51.829580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:51.829668 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:51.830039 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:51.830042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:51.830044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:51.830249 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:51.830365 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:51.830947 (+ 582us) server_negotiation.cc:299] Negotiation successful
0807 00:00:51.831087 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 00:00:51.832170 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:51.818931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:43099 (local address 127.8.204.65:60279)
0807 00:00:51.819401 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:51.819446 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:51.820694 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:51.821217 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:51.821224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:51.821577 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:51.822055 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:51.822069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.822899 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.822902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:51.823789 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:51.823798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:51.824265 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:51.824268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:51.824410 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:51.824982 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:51.825008 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:51.826901 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:51.829150 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:51.829156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:51.829170 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:51.829447 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:51.829769 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:51.829773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:51.829775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:51.829928 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:51.830364 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:51.830371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:51.830649 (+ 278us) client_negotiation.cc:769] Sending connection context
0807 00:00:51.830902 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 00:00:51.831274 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260807 00:00:51.833534 533 heartbeater.cc:347] Connected to a master server at 127.8.204.126:43099
I20260807 00:00:51.833812 533 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:51.834333 533 heartbeater.cc:511] Master 127.8.204.126:43099 requested a full tablet report, sending...
I20260807 00:00:51.836190 32736 ts_manager.cc:194] Registered new tserver with Master: cec3517ee27e4131b4006af4bae107c4 (127.8.204.65:39351)
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260807 00:00:51.837715 32736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:60279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260807 00:00:51.857450 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:51.976480 538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:51.976771 538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:51.976818 538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:51.980898 538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:51.980966 538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:51.981026 538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:51.986541 538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:51.987707 538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:51.988734 538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:51.999437 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:51.999437 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:51.999529 538 server_base.cc:1034] running on GCE node
W20260807 00:00:51.999430 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:52.000233 538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:52.000962 538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:52.002138 538 hybrid_clock.cc:648] HybridClock initialized: now 1786060852002120 us; error 34 us; skew 500 ppm
Aug 07 00:00:51 dist-test-slave-39l8 krb5kdc[32675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:52.005314 538 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260807 00:00:52.006337 538 webserver.cc:460] Webserver started at http://127.8.204.66:40887/ using document root <none> and password file <none>
I20260807 00:00:52.007030 538 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:52.007100 538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:52.007324 538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:52.009435 538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "9206e14334ea44f99fe18e59195b68bb"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "23c9fc44e2fcae87f6d183aa4b46ea08"
server_key_iv: "0ca7b9d0335a87fbb4527bdad1637a88"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.010020 538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "9206e14334ea44f99fe18e59195b68bb"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "23c9fc44e2fcae87f6d183aa4b46ea08"
server_key_iv: "0ca7b9d0335a87fbb4527bdad1637a88"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.022145 538 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.001s
I20260807 00:00:52.025264 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.026682 538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:00:52.026882 538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "9206e14334ea44f99fe18e59195b68bb"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "23c9fc44e2fcae87f6d183aa4b46ea08"
server_key_iv: "0ca7b9d0335a87fbb4527bdad1637a88"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.027014 538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:52.056391 538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:52.059624 538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:52.059855 538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:52.060539 538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:52.061436 538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:52.061486 538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.061560 538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:52.061597 538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.071322 538 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:43215
I20260807 00:00:52.071359 716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:43215 every 8 connection(s)
I20260807 00:00:52.072438 538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:52.076887 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 538
I20260807 00:00:52.077028 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:00:52.077314 9009 external_mini_cluster.cc:1428] Setting key 09e3d66ec8d684addcfba980616cc022
Aug 07 00:00:52 dist-test-slave-39l8 krb5kdc[32675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060851, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:52.088021 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.074578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:59007 (local address 127.8.204.126:43099)
0807 00:00:52.074785 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.074790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.075727 (+ 937us) server_negotiation.cc:407] Connection header received
0807 00:00:52.077205 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.077209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.077270 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.077382 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:52.078782 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.079301 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.080475 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.080711 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.080824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.083518 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:52.083522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:52.083524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:52.083547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:52.085155 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.085975 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.085979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.085980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.086065 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.086593 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.086595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.086596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.086842 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:52.086927 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.087750 (+ 823us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.087870 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:52.089396 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.075076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:43099 (local address 127.8.204.66:59007)
0807 00:00:52.075563 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.075608 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.076912 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.077563 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.077570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.077940 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:52.078611 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.078629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.079437 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.079441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.080338 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.080348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.080886 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.080890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.081067 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.081728 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:52.081745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:52.083334 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:52.085324 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:52.085335 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:52.085358 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:52.085839 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:52.086211 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:52.086217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:52.086222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:52.086463 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:52.086969 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:52.086979 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:52.087382 (+ 403us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.087655 (+ 273us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.088092 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260807 00:00:52.090930 717 heartbeater.cc:347] Connected to a master server at 127.8.204.126:43099
I20260807 00:00:52.091293 717 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:52.091871 717 heartbeater.cc:511] Master 127.8.204.126:43099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260807 00:00:52.093279 32736 ts_manager.cc:194] Registered new tserver with Master: 9206e14334ea44f99fe18e59195b68bb (127.8.204.66:43215)
I20260807 00:00:52.094054 32736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:59007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260807 00:00:52.110397 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:52.230019 725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:52.230352 725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:52.230439 725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:52.234637 725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:52.234776 725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:52.234877 725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:52.240384 725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:52.241607 725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:52.242763 725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:52.252924 731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:52.252933 736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:52.252939 730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:52.253392 725 server_base.cc:1034] running on GCE node
I20260807 00:00:52.253880 725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:52.254565 725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:52.255792 725 hybrid_clock.cc:648] HybridClock initialized: now 1786060852255763 us; error 39 us; skew 500 ppm
Aug 07 00:00:52 dist-test-slave-39l8 krb5kdc[32675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060852, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:52.259178 725 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260807 00:00:52.260212 725 webserver.cc:460] Webserver started at http://127.8.204.67:36707/ using document root <none> and password file <none>
I20260807 00:00:52.260879 725 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:52.261023 725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:52.261292 725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:52.263758 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "19255dfd00224f0b87370aae9dc8b24a"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "9a48b6f704afc40b6b43db1e6bedab23"
server_key_iv: "5657dc4490b84bb83ea3a08c32aba466"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.264420 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "19255dfd00224f0b87370aae9dc8b24a"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "9a48b6f704afc40b6b43db1e6bedab23"
server_key_iv: "5657dc4490b84bb83ea3a08c32aba466"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.268870 725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 00:00:52.271796 749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.273345 725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 00:00:52.273548 725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "19255dfd00224f0b87370aae9dc8b24a"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "9a48b6f704afc40b6b43db1e6bedab23"
server_key_iv: "5657dc4490b84bb83ea3a08c32aba466"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.273706 725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:52.287416 725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:52.290436 725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:52.290673 725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:52.291357 725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:52.292320 725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:52.292390 725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.292465 725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:52.292507 725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.302003 725 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:46175
I20260807 00:00:52.302034 912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:46175 every 8 connection(s)
I20260807 00:00:52.303042 725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:00:52.308996 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 725
I20260807 00:00:52.309151 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:00:52.309473 9009 external_mini_cluster.cc:1428] Setting key b0629cdd2e85ee214169f13441c78109
Aug 07 00:00:52 dist-test-slave-39l8 krb5kdc[32675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060852, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:52.318171 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.304924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:54829 (local address 127.8.204.126:43099)
0807 00:00:52.305067 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.305071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.306013 (+ 942us) server_negotiation.cc:407] Connection header received
0807 00:00:52.307428 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.307432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.307481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.307567 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:52.308741 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.309372 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.311027 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.311361 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.311473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.314313 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:52.314316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:52.314318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:52.314343 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:52.315857 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.316461 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.316465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.316467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.316551 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.316931 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.316935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.316937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.317141 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:52.317254 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.317859 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.317991 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:52.319113 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.305354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:43099 (local address 127.8.204.67:54829)
0807 00:00:52.305847 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.305891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.307197 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.307741 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.307747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.308101 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:52.308597 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.308611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.309513 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.309518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.310886 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.310898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.311597 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.311603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.311831 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.312551 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:52.312569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:52.314140 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:52.315988 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:52.315995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:52.316014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:52.316348 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:52.316661 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:52.316664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:52.316666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:52.316826 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:52.317254 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:52.317262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:52.317533 (+ 271us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.317803 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.318156 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260807 00:00:52.320325 914 heartbeater.cc:347] Connected to a master server at 127.8.204.126:43099
I20260807 00:00:52.320607 914 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:52.321142 914 heartbeater.cc:511] Master 127.8.204.126:43099 requested a full tablet report, sending...
I20260807 00:00:52.322330 32736 ts_manager.cc:194] Registered new tserver with Master: 19255dfd00224f0b87370aae9dc8b24a (127.8.204.67:46175)
I20260807 00:00:52.323087 32736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:54829
I20260807 00:00:52.323769 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:00:52.333551 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.325832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.8.204.126:43099)
0807 00:00:52.325984 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.325989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.326154 (+ 165us) server_negotiation.cc:407] Connection header received
0807 00:00:52.326242 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.326244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.326282 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.326344 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:52.327122 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.327649 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.328498 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.328704 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.328782 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.330088 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:52.330092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:52.330094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:52.330118 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:52.331741 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.332220 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.332223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.332225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.332318 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:52.332597 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:52.332600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:52.332602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:52.332824 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:52.333003 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.333224 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.333349 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 00:00:52.336609 32734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:00:52.339658 32734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:00:52.354231 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.348639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:39351 (local address 127.0.0.1:38434)
0807 00:00:52.349433 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.349484 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.349640 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.350179 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.350185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.350212 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:52.350375 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.350386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.351639 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.351644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.352833 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.352843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.353650 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.353653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.353770 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.353808 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.353948 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.354081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260807 00:00:52.354243 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.348966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.8.204.65:39351)
0807 00:00:52.349368 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.349375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.349576 (+ 201us) server_negotiation.cc:407] Connection header received
0807 00:00:52.349802 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.349808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.350027 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.350179 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:52.350515 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.351528 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.352976 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.353524 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.353649 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.353809 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.353978 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.354078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":101,"threads_started":1}
I20260807 00:00:52.355882 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.8.204.67:46175)
0807 00:00:52.349676 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.349690 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.350122 (+ 432us) server_negotiation.cc:407] Connection header received
0807 00:00:52.350200 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.350205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.350374 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.350529 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:52.350993 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.352249 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.354144 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.355003 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.355121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.355259 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.355616 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.355720 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":142,"threads_started":1}
I20260807 00:00:52.355996 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.349412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:43215 (local address 127.0.0.1:50162)
0807 00:00:52.350243 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.350256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.350337 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.350996 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.350999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.351026 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:52.351114 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.351119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.352443 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.352446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.353983 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.353995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.355132 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.355136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.355235 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.355260 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.355758 (+ 498us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.355831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":195,"thread_start_us":86,"threads_started":1}
I20260807 00:00:52.356408 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.349833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.8.204.66:43215)
0807 00:00:52.350576 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.350586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.350603 (+ 17us) server_negotiation.cc:407] Connection header received
0807 00:00:52.350664 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.350668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.350860 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.351011 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:52.351211 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.352305 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.354144 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.355003 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.355121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.355259 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.356161 (+ 902us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.356279 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":125,"threads_started":1}
I20260807 00:00:52.357213 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.348639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:46175 (local address 127.0.0.1:60130)
0807 00:00:52.349886 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.349899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.349977 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.350696 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.350699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.350718 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:52.350858 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.350865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.352424 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.352429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.353983 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.353995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.355337 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.355342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.355442 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.355460 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.355564 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.357115 (+ 1551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":173,"spinlock_wait_cycles":5504,"thread_start_us":97,"threads_started":1}
I20260807 00:00:52.358129 440 tablet_service.cc:1467] Processing CreateTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 (DEFAULT_TABLE table=test-table [id=5c723570b51f471d89e5720439014ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:52.359403 440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5302775a7f7848fe989a7dd7a14bcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:52.359091 631 tablet_service.cc:1467] Processing CreateTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 (DEFAULT_TABLE table=test-table [id=5c723570b51f471d89e5720439014ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:52.359537 823 tablet_service.cc:1467] Processing CreateTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 (DEFAULT_TABLE table=test-table [id=5c723570b51f471d89e5720439014ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:52.360105 631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5302775a7f7848fe989a7dd7a14bcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:52.360455 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5302775a7f7848fe989a7dd7a14bcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:52.366557 937 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Bootstrap starting.
I20260807 00:00:52.367959 938 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Bootstrap starting.
I20260807 00:00:52.368919 937 tablet_bootstrap.cc:654] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:52.369832 938 tablet_bootstrap.cc:654] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:52.370179 937 log.cc:824] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:52.370286 940 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Bootstrap starting.
I20260807 00:00:52.371454 938 log.cc:824] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:52.372677 937 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: No bootstrap required, opened a new log
I20260807 00:00:52.372627 940 tablet_bootstrap.cc:654] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:52.372897 937 ts_tablet_manager.cc:1397] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 00:00:52.373478 938 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: No bootstrap required, opened a new log
I20260807 00:00:52.373674 938 ts_tablet_manager.cc:1397] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:52.374027 940 log.cc:824] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:52.376134 940 tablet_bootstrap.cc:492] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: No bootstrap required, opened a new log
I20260807 00:00:52.376338 940 ts_tablet_manager.cc:1397] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:52.377501 937 raft_consensus.cc:348] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.377683 938 raft_consensus.cc:348] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.377924 937 raft_consensus.cc:374] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:52.378010 937 raft_consensus.cc:729] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19255dfd00224f0b87370aae9dc8b24a, State: Initialized, Role: FOLLOWER
I20260807 00:00:52.378082 938 raft_consensus.cc:374] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:52.378196 938 raft_consensus.cc:729] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec3517ee27e4131b4006af4bae107c4, State: Initialized, Role: FOLLOWER
I20260807 00:00:52.378566 937 consensus_queue.cc:260] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.378616 938 consensus_queue.cc:260] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.379340 914 heartbeater.cc:503] Master 127.8.204.126:43099 was elected leader, sending a full tablet report...
I20260807 00:00:52.379653 938 ts_tablet_manager.cc:1428] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 00:00:52.380005 533 heartbeater.cc:503] Master 127.8.204.126:43099 was elected leader, sending a full tablet report...
I20260807 00:00:52.380494 937 ts_tablet_manager.cc:1428] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260807 00:00:52.380370 940 raft_consensus.cc:348] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.380673 940 raft_consensus.cc:374] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:52.380739 940 raft_consensus.cc:729] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9206e14334ea44f99fe18e59195b68bb, State: Initialized, Role: FOLLOWER
I20260807 00:00:52.381174 940 consensus_queue.cc:260] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.381880 717 heartbeater.cc:503] Master 127.8.204.126:43099 was elected leader, sending a full tablet report...
I20260807 00:00:52.382179 940 ts_tablet_manager.cc:1428] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260807 00:00:52.554082 915 tablet.cc:2369] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 00:00:52.567996 534 tablet.cc:2369] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 00:00:52.573660 719 tablet.cc:2369] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:52.596530 945 raft_consensus.cc:482] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:52.596742 945 raft_consensus.cc:504] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.597899 945 leader_election.cc:288] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec3517ee27e4131b4006af4bae107c4 (127.8.204.65:39351), 9206e14334ea44f99fe18e59195b68bb (127.8.204.66:43215)
I20260807 00:00:52.603528 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.598331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:50993 (local address 127.8.204.66:43215)
0807 00:00:52.598483 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.598489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.598862 (+ 373us) server_negotiation.cc:407] Connection header received
0807 00:00:52.598949 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.598952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.599019 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.599095 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:52.599380 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.600138 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.601961 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.602610 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.602772 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.602829 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.603298 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.603393 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 00:00:52.603543 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.598386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:56005 (local address 127.8.204.65:39351)
0807 00:00:52.598541 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.598548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.598566 (+ 18us) server_negotiation.cc:407] Connection header received
0807 00:00:52.598790 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.598793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.598865 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.598947 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:52.599271 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.600094 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.601961 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.602623 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.602772 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.602829 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.603298 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.603393 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 00:00:52.603523 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.598235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:39351 (local address 127.8.204.67:56005)
0807 00:00:52.598492 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.598509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.598629 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.598987 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.598989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.599003 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:52.599147 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.599154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.600239 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.600243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.601800 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.601811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.602774 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.602778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.602980 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.603140 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.603227 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.603335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":206}
I20260807 00:00:52.603531 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.598235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:43215 (local address 127.8.204.67:50993)
0807 00:00:52.598737 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.598754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:52.598867 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:52.599123 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:52.599126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:52.599140 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:52.599261 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.599268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.600239 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.600243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.601799 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:52.601811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.602774 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.602778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:52.602982 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.603145 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 00:00:52.603227 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 00:00:52.603335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":356,"thread_start_us":121,"threads_started":1}
I20260807 00:00:52.604353 658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5302775a7f7848fe989a7dd7a14bcf25" candidate_uuid: "19255dfd00224f0b87370aae9dc8b24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9206e14334ea44f99fe18e59195b68bb" is_pre_election: true
I20260807 00:00:52.604516 467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5302775a7f7848fe989a7dd7a14bcf25" candidate_uuid: "19255dfd00224f0b87370aae9dc8b24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3517ee27e4131b4006af4bae107c4" is_pre_election: true
I20260807 00:00:52.604725 658 raft_consensus.cc:2454] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19255dfd00224f0b87370aae9dc8b24a in term 0.
I20260807 00:00:52.604779 467 raft_consensus.cc:2454] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19255dfd00224f0b87370aae9dc8b24a in term 0.
I20260807 00:00:52.605219 753 leader_election.cc:302] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19255dfd00224f0b87370aae9dc8b24a, 9206e14334ea44f99fe18e59195b68bb; no voters:
I20260807 00:00:52.605552 945 raft_consensus.cc:2781] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:00:52.605648 945 raft_consensus.cc:482] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:00:52.605684 945 raft_consensus.cc:3037] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:52.606805 945 raft_consensus.cc:504] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.607189 945 leader_election.cc:288] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [CANDIDATE]: Term 1 election: Requested vote from peers cec3517ee27e4131b4006af4bae107c4 (127.8.204.65:39351), 9206e14334ea44f99fe18e59195b68bb (127.8.204.66:43215)
I20260807 00:00:52.607568 467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5302775a7f7848fe989a7dd7a14bcf25" candidate_uuid: "19255dfd00224f0b87370aae9dc8b24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3517ee27e4131b4006af4bae107c4"
I20260807 00:00:52.607656 658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5302775a7f7848fe989a7dd7a14bcf25" candidate_uuid: "19255dfd00224f0b87370aae9dc8b24a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9206e14334ea44f99fe18e59195b68bb"
I20260807 00:00:52.607717 467 raft_consensus.cc:3037] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:52.607836 658 raft_consensus.cc:3037] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:52.608860 467 raft_consensus.cc:2454] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19255dfd00224f0b87370aae9dc8b24a in term 1.
I20260807 00:00:52.608908 658 raft_consensus.cc:2454] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19255dfd00224f0b87370aae9dc8b24a in term 1.
I20260807 00:00:52.609223 752 leader_election.cc:302] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19255dfd00224f0b87370aae9dc8b24a, cec3517ee27e4131b4006af4bae107c4; no voters:
I20260807 00:00:52.609414 945 raft_consensus.cc:2781] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:52.609670 945 raft_consensus.cc:686] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 1 LEADER]: Becoming Leader. State: Replica: 19255dfd00224f0b87370aae9dc8b24a, State: Running, Role: LEADER
I20260807 00:00:52.610054 945 consensus_queue.cc:237] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } }
I20260807 00:00:52.613916 32734 catalog_manager.cc:5495] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a reported cstate change: term changed from 0 to 1, leader changed from <none> to 19255dfd00224f0b87370aae9dc8b24a (127.8.204.67). New cstate: current_term: 1 leader_uuid: "19255dfd00224f0b87370aae9dc8b24a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19255dfd00224f0b87370aae9dc8b24a" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 } health_report { overall_health: UNKNOWN } } }
I20260807 00:00:52.648830 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.645158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.8.204.67:46175)
0807 00:00:52.645338 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.645343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.645363 (+ 20us) server_negotiation.cc:407] Connection header received
0807 00:00:52.645441 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.645443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.645497 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.645620 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:52.645895 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.646407 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.647304 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.647522 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.647632 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.647972 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:52.648408 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:00:52.648551 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.648592 (+ 41us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.648648 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 00:00:52.654991 658 raft_consensus.cc:1261] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 1 FOLLOWER]: Refusing update from remote peer 19255dfd00224f0b87370aae9dc8b24a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:52.655030 467 raft_consensus.cc:1261] T 5302775a7f7848fe989a7dd7a14bcf25 P cec3517ee27e4131b4006af4bae107c4 [term 1 FOLLOWER]: Refusing update from remote peer 19255dfd00224f0b87370aae9dc8b24a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:52.655750 945 consensus_queue.cc:1035] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9206e14334ea44f99fe18e59195b68bb" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:52.655943 950 consensus_queue.cc:1035] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec3517ee27e4131b4006af4bae107c4" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:52.665752 957 mvcc.cc:205] Tried to move back new op lower bound from 7315705252467200000 to 7315705252293607424. Current Snapshot: MvccSnapshot[applied={T|T < 7315705252467200000}]
I20260807 00:00:52.674314 32736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:52.674525 32736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:52.677263 32736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70183f60b1c24c51b2c5886c28046bf3: Sending DeleteTablet for 3 replicas of tablet 5302775a7f7848fe989a7dd7a14bcf25
I20260807 00:00:52.678221 440 tablet_service.cc:1514] Processing DeleteTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:38434
I20260807 00:00:52.678265 631 tablet_service.cc:1514] Processing DeleteTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:50162
I20260807 00:00:52.678834 823 tablet_service.cc:1514] Processing DeleteTablet for tablet 5302775a7f7848fe989a7dd7a14bcf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:52 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:60130
I20260807 00:00:52.679141 971 tablet_replica.cc:330] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: stopping tablet replica
I20260807 00:00:52.679507 971 raft_consensus.cc:2229] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:52.679682 973 tablet_replica.cc:330] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: stopping tablet replica
I20260807 00:00:52.679890 971 raft_consensus.cc:2258] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:52.679935 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 352
I20260807 00:00:52.679975 973 raft_consensus.cc:2229] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 1 LEADER]: Raft consensus shutting down.
I20260807 00:00:52.681226 973 raft_consensus.cc:2258] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:52.683650 973 ts_tablet_manager.cc:1905] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:52.686954 973 ts_tablet_manager.cc:1918] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:52.687057 973 log.cc:1196] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/5302775a7f7848fe989a7dd7a14bcf25
I20260807 00:00:52.687376 973 ts_tablet_manager.cc:1939] T 5302775a7f7848fe989a7dd7a14bcf25 P 19255dfd00224f0b87370aae9dc8b24a: Deleting consensus metadata
I20260807 00:00:52.690230 971 ts_tablet_manager.cc:1905] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:52.693400 971 ts_tablet_manager.cc:1918] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:52.693503 971 log.cc:1196] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/5302775a7f7848fe989a7dd7a14bcf25
I20260807 00:00:52.693804 971 ts_tablet_manager.cc:1939] T 5302775a7f7848fe989a7dd7a14bcf25 P 9206e14334ea44f99fe18e59195b68bb: Deleting consensus metadata
I20260807 00:00:52.699584 32713 catalog_manager.cc:4841] TS 19255dfd00224f0b87370aae9dc8b24a (127.8.204.67:46175): tablet 5302775a7f7848fe989a7dd7a14bcf25 (table test-table [id=5c723570b51f471d89e5720439014ecb]) successfully deleted
I20260807 00:00:52.699795 32713 catalog_manager.cc:4841] TS 9206e14334ea44f99fe18e59195b68bb (127.8.204.66:43215): tablet 5302775a7f7848fe989a7dd7a14bcf25 (table test-table [id=5c723570b51f471d89e5720439014ecb]) successfully deleted
W20260807 00:00:52.707157 32711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:39351 (error 108)
I20260807 00:00:52.707361 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 538
I20260807 00:00:52.711988 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.711617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:39351 (local address 127.0.0.1:38436)
0807 00:00:52.711751 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:52.711847 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 00:00:52.712221 32711 catalog_manager.cc:4584] TS cec3517ee27e4131b4006af4bae107c4 (127.8.204.65:39351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5302775a7f7848fe989a7dd7a14bcf25: Network error: Client connection negotiation failed: client connection to 127.8.204.65:39351: connect: Connection refused (error 111)
I20260807 00:00:52.720893 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 725
I20260807 00:00:52.736065 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 32692
2026-08-07T00:00:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2691 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T00:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:00:52Z Disabled control of system clock
I20260807 00:00:52.773757 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44641
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:00:52.887126 979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:52.887418 979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:52.887454 979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:52.891500 979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:52.891562 979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:52.891578 979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:52.891595 979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:52.891611 979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:52.896394 979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:52.897485 979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:52.898844 979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:52.907526 985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:52.907588 987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:52.907738 979 server_base.cc:1034] running on GCE node
W20260807 00:00:52.907588 989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:52.908514 979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:52.909655 979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:52.910862 979 hybrid_clock.cc:648] HybridClock initialized: now 1786060852910849 us; error 27 us; skew 500 ppm
I20260807 00:00:52.912757 979 webserver.cc:460] Webserver started at http://127.8.204.126:34763/ using document root <none> and password file <none>
I20260807 00:00:52.913371 979 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:52.913465 979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:52.913779 979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:52.915969 979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "eea24315c8294831abd0eab3fd9d3a25"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "f92f7cc8bb8221e669947f4ce85e4c67"
server_key_iv: "58664e310a0211d9b26bb6ee62fab51e"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.916503 979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "eea24315c8294831abd0eab3fd9d3a25"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "f92f7cc8bb8221e669947f4ce85e4c67"
server_key_iv: "58664e310a0211d9b26bb6ee62fab51e"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.920665 979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 00:00:52.923197 995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:52.924464 979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:52.924587 979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "eea24315c8294831abd0eab3fd9d3a25"
format_stamp: "Formatted at 2026-08-07 00:00:52 on dist-test-slave-39l8"
server_key: "f92f7cc8bb8221e669947f4ce85e4c67"
server_key_iv: "58664e310a0211d9b26bb6ee62fab51e"
server_key_version: "encryptionkey@0"
I20260807 00:00:52.924681 979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:52.952844 979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:52.953552 979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:52.953727 979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:52.961413 979 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44641
I20260807 00:00:52.961421 1070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44641 every 8 connection(s)
I20260807 00:00:52.962442 979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:52.965835 1071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:52.970423 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 979
I20260807 00:00:52.970538 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:00:52.970899 9009 external_mini_cluster.cc:1428] Setting key d30556e291a80bcc43be5566c274664d
I20260807 00:00:52.972101 1071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: Bootstrap starting.
I20260807 00:00:52.974512 1071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:52.975426 1071 log.cc:824] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:52.977837 1071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: No bootstrap required, opened a new log
I20260807 00:00:52.980914 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:52.972457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.8.204.126:44641)
0807 00:00:52.973094 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:52.973111 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:52.973146 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:00:52.973803 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:52.973821 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:52.974218 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:52.974494 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:52.974975 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.976106 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.977393 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:52.977864 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:52.978055 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:52.978957 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:52.978962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:52.978975 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:52.979365 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:52.979422 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:52.979441 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:52.979520 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:52.979702 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 00:00:52.979998 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20260807 00:00:52.981480 1071 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } }
I20260807 00:00:52.981736 1071 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:52.981832 1071 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea24315c8294831abd0eab3fd9d3a25, State: Initialized, Role: FOLLOWER
I20260807 00:00:52.982369 1071 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } }
I20260807 00:00:52.982542 1071 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:52.982613 1071 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:52.982733 1071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:52.983922 1071 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } }
I20260807 00:00:52.984254 1071 leader_election.cc:302] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea24315c8294831abd0eab3fd9d3a25; no voters:
I20260807 00:00:52.984524 1071 leader_election.cc:288] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:00:52.984649 1078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:52.984899 1078 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [term 1 LEADER]: Becoming Leader. State: Replica: eea24315c8294831abd0eab3fd9d3a25, State: Running, Role: LEADER
I20260807 00:00:52.985210 1078 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } }
I20260807 00:00:52.985872 1071 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:52.986980 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea24315c8294831abd0eab3fd9d3a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } } }
I20260807 00:00:52.987104 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:52.987630 1080 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea24315c8294831abd0eab3fd9d3a25. Latest consensus state: current_term: 1 leader_uuid: "eea24315c8294831abd0eab3fd9d3a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea24315c8294831abd0eab3fd9d3a25" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44641 } } }
I20260807 00:00:52.987787 1080 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:52.988581 1090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:52.991753 1090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:52.997453 1090 catalog_manager.cc:1347] Generated new cluster ID: fd220f0b0cc14cd797788cd7cc30b45e
I20260807 00:00:52.997534 1090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:53.008581 1090 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:00:53.009892 1090 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:53.021070 1090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: Generated new TSK 0
I20260807 00:00:53.021981 1090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:00:53.032722 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--builtin_ntp_servers=127.8.204.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:00:53.155032 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:53.155320 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:53.155360 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:53.159525 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:53.159593 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:53.159651 1103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:53.164830 1103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:53.166054 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:53.167614 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:53.177872 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:53.177872 1113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:53.178164 1103 server_base.cc:1034] running on GCE node
W20260807 00:00:53.177930 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:53.178882 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:53.179836 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:53.181037 1103 hybrid_clock.cc:648] HybridClock initialized: now 1786060853181002 us; error 47 us; skew 500 ppm
I20260807 00:00:53.183257 1103 webserver.cc:460] Webserver started at http://127.8.204.65:33523/ using document root <none> and password file <none>
I20260807 00:00:53.183887 1103 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:53.183954 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:53.184208 1103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:53.186466 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "e3879107b10549ec9ec83844f0366ae5"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "4f625989af4addc369f3d5b3bd107fb5"
server_key_iv: "afb49d85aef2e81cdcd2d2c94664ccd7"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.187162 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "e3879107b10549ec9ec83844f0366ae5"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "4f625989af4addc369f3d5b3bd107fb5"
server_key_iv: "afb49d85aef2e81cdcd2d2c94664ccd7"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.191143 1103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:00:53.193616 1122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.194952 1103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:53.195128 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "e3879107b10549ec9ec83844f0366ae5"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "4f625989af4addc369f3d5b3bd107fb5"
server_key_iv: "afb49d85aef2e81cdcd2d2c94664ccd7"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.195245 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:53.255568 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:53.256374 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:53.256598 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:53.257277 1103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:53.258333 1103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:53.258410 1103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.258486 1103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:53.258527 1103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.268343 1103 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:41501
I20260807 00:00:53.268376 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:41501 every 8 connection(s)
I20260807 00:00:53.269310 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:53.271574 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 1103
I20260807 00:00:53.271715 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:00:53.272049 9009 external_mini_cluster.cc:1428] Setting key 654873a38560f7e943d9ff99973a559f
I20260807 00:00:53.274093 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--builtin_ntp_servers=127.8.204.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:00:53.280246 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.271420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:46651 (local address 127.8.204.126:44641)
0807 00:00:53.271607 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.271612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.273055 (+ 1443us) server_negotiation.cc:407] Connection header received
0807 00:00:53.274746 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.274753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.274821 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.274923 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:53.275975 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.276590 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.277908 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.278173 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.278306 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.279668 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:53.279672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:53.279675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:53.279699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:53.279720 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:53.279729 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:53.279814 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.280009 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.280081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 00:00:53.281209 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.271965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44641 (local address 127.8.204.65:46651)
0807 00:00:53.272874 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.272933 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.274467 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.275104 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.275112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.275205 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:53.275808 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.275822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.276754 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.276757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.277761 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.277772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.278357 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.278362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.278550 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.279343 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:53.279345 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:53.279390 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:53.279465 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:53.279831 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:53.279842 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:53.279851 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.280003 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.280317 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":170,"threads_started":1}
I20260807 00:00:53.282997 1288 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44641
I20260807 00:00:53.283336 1288 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:53.283922 1288 heartbeater.cc:511] Master 127.8.204.126:44641 requested a full tablet report, sending...
I20260807 00:00:53.285945 1016 ts_manager.cc:194] Registered new tserver with Master: e3879107b10549ec9ec83844f0366ae5 (127.8.204.65:41501)
I20260807 00:00:53.288004 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.65:46651
W20260807 00:00:53.401993 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:53.402280 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:53.402321 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:53.406467 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:53.406545 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:53.406605 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:53.411552 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:53.412724 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:53.414149 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:53.424188 1304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:53.424191 1306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:53.424224 1303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:53.424549 1291 server_base.cc:1034] running on GCE node
I20260807 00:00:53.425179 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:53.425889 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:53.427129 1291 hybrid_clock.cc:648] HybridClock initialized: now 1786060853427102 us; error 33 us; skew 500 ppm
I20260807 00:00:53.429448 1291 webserver.cc:460] Webserver started at http://127.8.204.66:42607/ using document root <none> and password file <none>
I20260807 00:00:53.430167 1291 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:53.430280 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:53.430565 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:53.433110 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "b7cda1228f3a4754b15b2e38be55df39"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "cfa7870f571d8333eb7dc6345ef025ff"
server_key_iv: "99529998f5ff3fc1ba0044ee31ad9985"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.433786 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "b7cda1228f3a4754b15b2e38be55df39"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "cfa7870f571d8333eb7dc6345ef025ff"
server_key_iv: "99529998f5ff3fc1ba0044ee31ad9985"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.438153 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 00:00:53.441237 1313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.442694 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:00:53.442935 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "b7cda1228f3a4754b15b2e38be55df39"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "cfa7870f571d8333eb7dc6345ef025ff"
server_key_iv: "99529998f5ff3fc1ba0044ee31ad9985"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.443075 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:53.470917 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:53.471761 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:53.472015 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:53.472769 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:53.473825 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:53.473904 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.473986 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:53.474041 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.483780 1291 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:38913
I20260807 00:00:53.483819 1470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:38913 every 8 connection(s)
I20260807 00:00:53.484774 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:53.493018 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 1291
I20260807 00:00:53.493142 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:00:53.493448 9009 external_mini_cluster.cc:1428] Setting key e58dad257d37a919c157ec1e74da0fd5
I20260807 00:00:53.496081 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--builtin_ntp_servers=127.8.204.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:00:53.496462 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.487169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:54033 (local address 127.8.204.126:44641)
0807 00:00:53.487321 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.487325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.488711 (+ 1386us) server_negotiation.cc:407] Connection header received
0807 00:00:53.490363 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.490367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.490431 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.490535 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:53.491618 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.492229 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.493680 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.494008 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.494148 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.495809 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:53.495812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:53.495814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:53.495828 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:53.495848 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:53.495856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:53.495916 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.496178 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.496283 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:00:53.497649 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.487632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44641 (local address 127.8.204.66:54033)
0807 00:00:53.488500 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.488555 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.490104 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.490797 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.490805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.490902 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:53.491442 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.491456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.492395 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.492399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.493529 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.493556 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.494248 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.494253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.494467 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.495432 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:53.495435 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:53.495494 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:53.495592 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:53.495953 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:53.495968 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:53.495978 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.496134 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.496491 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":126,"threads_started":1}
I20260807 00:00:53.499251 1471 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44641
I20260807 00:00:53.499630 1471 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:53.500237 1471 heartbeater.cc:511] Master 127.8.204.126:44641 requested a full tablet report, sending...
I20260807 00:00:53.501536 1016 ts_manager.cc:194] Registered new tserver with Master: b7cda1228f3a4754b15b2e38be55df39 (127.8.204.66:38913)
I20260807 00:00:53.502174 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.66:54033
W20260807 00:00:53.622622 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:53.622967 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:53.623047 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:53.627238 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:53.627298 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:53.627398 1476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:53.632503 1476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:53.633595 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:53.635056 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:53.644985 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:53.644979 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:53.644985 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:53.645298 1476 server_base.cc:1034] running on GCE node
I20260807 00:00:53.645824 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:53.648095 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:53.649307 1476 hybrid_clock.cc:648] HybridClock initialized: now 1786060853649270 us; error 48 us; skew 500 ppm
I20260807 00:00:53.657409 1476 webserver.cc:460] Webserver started at http://127.8.204.67:40195/ using document root <none> and password file <none>
I20260807 00:00:53.658129 1476 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:53.658241 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:53.658514 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:53.660992 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "045229109571869fa9db87efc2ca0f4d"
server_key_iv: "8083bcb64018f7b4e2f8faa0e96e61da"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.661653 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "045229109571869fa9db87efc2ca0f4d"
server_key_iv: "8083bcb64018f7b4e2f8faa0e96e61da"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.665834 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 00:00:53.668591 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.669952 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:53.670116 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc"
format_stamp: "Formatted at 2026-08-07 00:00:53 on dist-test-slave-39l8"
server_key: "045229109571869fa9db87efc2ca0f4d"
server_key_iv: "8083bcb64018f7b4e2f8faa0e96e61da"
server_key_version: "encryptionkey@0"
I20260807 00:00:53.670251 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:53.694427 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:53.695389 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:53.695626 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:53.696341 1476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:53.697425 1476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:53.697507 1476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.697588 1476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:53.697642 1476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:53.708238 1476 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:44607
I20260807 00:00:53.708273 1651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:44607 every 8 connection(s)
I20260807 00:00:53.709254 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:00:53.716507 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 1476
I20260807 00:00:53.716688 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:00:53.716993 9009 external_mini_cluster.cc:1428] Setting key 2e78033abf5bacb583f1adc5e8e02567
I20260807 00:00:53.719827 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.711352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:46577 (local address 127.8.204.126:44641)
0807 00:00:53.711520 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.711524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.712679 (+ 1155us) server_negotiation.cc:407] Connection header received
0807 00:00:53.714222 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.714225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.714286 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.714397 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:53.715431 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.716001 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.717617 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.717932 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.718085 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.719344 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:53.719348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:53.719351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:53.719367 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:53.719392 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:53.719403 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:53.719486 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.719650 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.719706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 00:00:53.720758 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.711804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44641 (local address 127.8.204.67:46577)
0807 00:00:53.712508 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.712550 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.714004 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.714573 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.714580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.714663 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:53.715256 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.715279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.716158 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.716161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.717432 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.717444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.718098 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.718102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.718291 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.719001 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:53.719002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:53.719045 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:53.719140 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:53.719495 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:53.719504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:53.719509 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.719629 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.719856 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":150,"threads_started":1}
I20260807 00:00:53.722043 1652 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44641
I20260807 00:00:53.722318 1652 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:53.722950 1652 heartbeater.cc:511] Master 127.8.204.126:44641 requested a full tablet report, sending...
I20260807 00:00:53.724037 1016 ts_manager.cc:194] Registered new tserver with Master: d7ee41cf2916478b9b9cb3007ce9f4dc (127.8.204.67:44607)
I20260807 00:00:53.724666 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.67:46577
I20260807 00:00:53.731382 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:00:53.737568 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.733621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.8.204.126:44641)
0807 00:00:53.733772 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.733777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.733971 (+ 194us) server_negotiation.cc:407] Connection header received
0807 00:00:53.734081 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.734083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.734127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.734195 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:53.734473 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.735067 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.735972 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.736209 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.736292 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.737099 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:53.737102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:53.737105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:53.737127 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:53.737147 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:53.737156 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:53.737232 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.737312 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.737363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 00:00:53.740700 1016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:00:53.743908 1016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:00:53.759745 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.753134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41501 (local address 127.0.0.1:41068)
0807 00:00:53.753642 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.753691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.753829 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.755213 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.755220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.755245 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.755406 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.755413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.756483 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.756490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.758036 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.758048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.759257 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.759261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.759350 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.759381 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.759457 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.759549 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":5,"thread_start_us":91,"threads_started":1}
I20260807 00:00:53.759804 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.753339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.8.204.65:41501)
0807 00:00:53.754699 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.754733 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.754770 (+ 37us) server_negotiation.cc:407] Connection header received
0807 00:00:53.754844 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.754848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.755023 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.755221 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.755534 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.756352 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.758208 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.759095 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.759252 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.759455 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.759513 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.759626 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":134,"threads_started":1}
I20260807 00:00:53.759930 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.753344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:38913 (local address 127.0.0.1:43464)
0807 00:00:53.753849 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.753864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.753939 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.755307 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.755310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.755345 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.755627 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.755632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.757123 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.757130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.758477 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.758488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.759564 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.759568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.759653 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.759675 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.759724 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.759793 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":246,"spinlock_wait_cycles":896,"thread_start_us":138,"threads_started":1}
I20260807 00:00:53.760222 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.754293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.8.204.67:44607)
0807 00:00:53.754804 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.754816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.754841 (+ 25us) server_negotiation.cc:407] Connection header received
0807 00:00:53.754919 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.754924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.755138 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.755812 (+ 674us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.755946 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.757020 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.758161 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.758870 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.758979 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.759186 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.759655 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.759760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":162,"threads_started":1}
I20260807 00:00:53.760795 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.753908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:44607 (local address 127.0.0.1:59068)
0807 00:00:53.754174 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.754191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.754405 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.755307 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.755310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.755346 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.755461 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.755468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.757150 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.757152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.758029 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.758040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.759003 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.759007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.759145 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.759219 (+ 74us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.759337 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.759474 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":72,"spinlock_wait_cycles":8960,"thread_start_us":107,"threads_started":1}
I20260807 00:00:53.761785 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.754387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.8.204.66:38913)
0807 00:00:53.754804 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.754817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.754841 (+ 24us) server_negotiation.cc:407] Connection header received
0807 00:00:53.754919 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.754924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.755138 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.755422 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.755737 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.756954 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.758623 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.759420 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.760579 (+ 1159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.760758 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.760809 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.761631 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":170,"threads_started":1}
I20260807 00:00:53.763264 1195 tablet_service.cc:1467] Processing CreateTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 (DEFAULT_TABLE table=test-table [id=cc749a17b8754fd4accf927d5b0191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:53.763756 1562 tablet_service.cc:1467] Processing CreateTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 (DEFAULT_TABLE table=test-table [id=cc749a17b8754fd4accf927d5b0191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:53.764434 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddf9b4abbfd4730b84f5b9fc33ab711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:53.764917 1562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddf9b4abbfd4730b84f5b9fc33ab711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:53.764632 1380 tablet_service.cc:1467] Processing CreateTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 (DEFAULT_TABLE table=test-table [id=cc749a17b8754fd4accf927d5b0191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:53.765722 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cddf9b4abbfd4730b84f5b9fc33ab711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:53.771044 1680 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: Bootstrap starting.
I20260807 00:00:53.772872 1681 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Bootstrap starting.
I20260807 00:00:53.773605 1680 tablet_bootstrap.cc:654] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:53.774025 1682 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Bootstrap starting.
I20260807 00:00:53.774972 1680 log.cc:824] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:53.775228 1681 tablet_bootstrap.cc:654] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:53.775988 1681 log.cc:824] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:53.776631 1682 tablet_bootstrap.cc:654] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:53.777252 1680 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: No bootstrap required, opened a new log
I20260807 00:00:53.777429 1682 log.cc:824] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:53.777465 1680 ts_tablet_manager.cc:1397] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 00:00:53.778108 1681 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: No bootstrap required, opened a new log
I20260807 00:00:53.778314 1681 ts_tablet_manager.cc:1397] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:53.779877 1682 tablet_bootstrap.cc:492] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: No bootstrap required, opened a new log
I20260807 00:00:53.780102 1682 ts_tablet_manager.cc:1397] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:53.782055 1680 raft_consensus.cc:348] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.782362 1680 raft_consensus.cc:374] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:53.782424 1680 raft_consensus.cc:729] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3879107b10549ec9ec83844f0366ae5, State: Initialized, Role: FOLLOWER
I20260807 00:00:53.782614 1681 raft_consensus.cc:348] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.782876 1681 raft_consensus.cc:374] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:53.782928 1681 raft_consensus.cc:729] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ee41cf2916478b9b9cb3007ce9f4dc, State: Initialized, Role: FOLLOWER
I20260807 00:00:53.782948 1680 consensus_queue.cc:260] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.783403 1681 consensus_queue.cc:260] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.783763 1288 heartbeater.cc:503] Master 127.8.204.126:44641 was elected leader, sending a full tablet report...
I20260807 00:00:53.784058 1680 ts_tablet_manager.cc:1428] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 00:00:53.784050 1682 raft_consensus.cc:348] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.784329 1681 ts_tablet_manager.cc:1428] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 00:00:53.784364 1682 raft_consensus.cc:374] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:53.784422 1682 raft_consensus.cc:729] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cda1228f3a4754b15b2e38be55df39, State: Initialized, Role: FOLLOWER
I20260807 00:00:53.784466 1652 heartbeater.cc:503] Master 127.8.204.126:44641 was elected leader, sending a full tablet report...
I20260807 00:00:53.784886 1682 consensus_queue.cc:260] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.785810 1471 heartbeater.cc:503] Master 127.8.204.126:44641 was elected leader, sending a full tablet report...
I20260807 00:00:53.785946 1682 ts_tablet_manager.cc:1428] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 00:00:53.895592 1686 raft_consensus.cc:482] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:53.895879 1686 raft_consensus.cc:504] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.897207 1686 leader_election.cc:288] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7cda1228f3a4754b15b2e38be55df39 (127.8.204.66:38913), d7ee41cf2916478b9b9cb3007ce9f4dc (127.8.204.67:44607)
I20260807 00:00:53.898916 1687 raft_consensus.cc:482] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:53.899101 1687 raft_consensus.cc:504] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.900377 1687 leader_election.cc:288] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3879107b10549ec9ec83844f0366ae5 (127.8.204.65:41501), b7cda1228f3a4754b15b2e38be55df39 (127.8.204.66:38913)
I20260807 00:00:53.902262 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.897442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:38913 (local address 127.8.204.65:33351)
0807 00:00:53.897837 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.897851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.898007 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.898373 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.898379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.898411 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.898599 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.898608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.899651 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.899655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.900837 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.900851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.901787 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.901791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.901889 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.901916 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.901991 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.902064 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":21,"thread_start_us":139,"threads_started":1}
I20260807 00:00:53.902436 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.898059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:60743 (local address 127.8.204.67:44607)
0807 00:00:53.898203 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.898206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.898357 (+ 151us) server_negotiation.cc:407] Connection header received
0807 00:00:53.898452 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.898455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.898509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.898598 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.898969 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.899728 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.901200 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.901774 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.901858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.901903 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.902229 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.902319 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:53.902781 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.897595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:33351 (local address 127.8.204.66:38913)
0807 00:00:53.897772 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.897778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.897944 (+ 166us) server_negotiation.cc:407] Connection header received
0807 00:00:53.898160 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.898165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.898225 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.898317 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.898757 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.899539 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.900946 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.901531 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.901643 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.901706 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.902539 (+ 833us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.902627 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 00:00:53.902832 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.897932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:44607 (local address 127.8.204.65:60743)
0807 00:00:53.898262 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.898276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.898367 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.898674 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.898677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.898696 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.898841 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.898849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.899831 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.899834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.901054 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.901065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.901996 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.902000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.902085 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.902103 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.902607 (+ 504us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.902673 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":25,"thread_start_us":98,"threads_started":1}
I20260807 00:00:53.903379 1407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "e3879107b10549ec9ec83844f0366ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cda1228f3a4754b15b2e38be55df39" is_pre_election: true
I20260807 00:00:53.903378 1589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "e3879107b10549ec9ec83844f0366ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" is_pre_election: true
I20260807 00:00:53.903609 1589 raft_consensus.cc:2454] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3879107b10549ec9ec83844f0366ae5 in term 0.
I20260807 00:00:53.903656 1407 raft_consensus.cc:2454] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3879107b10549ec9ec83844f0366ae5 in term 0.
I20260807 00:00:53.904134 1124 leader_election.cc:302] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7ee41cf2916478b9b9cb3007ce9f4dc, e3879107b10549ec9ec83844f0366ae5; no voters:
I20260807 00:00:53.904448 1686 raft_consensus.cc:2781] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:00:53.904615 1686 raft_consensus.cc:482] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:00:53.904748 1686 raft_consensus.cc:3037] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:53.905769 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.900553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41501 (local address 127.8.204.67:44615)
0807 00:00:53.900682 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.900703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.900783 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.901968 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.901970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.901982 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.902074 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.902082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.903127 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.903130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.904364 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.904375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.905361 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.905364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.905464 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.905495 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.905572 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.905639 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":56}
I20260807 00:00:53.905787 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.901592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:44615 (local address 127.8.204.65:41501)
0807 00:00:53.901701 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.901705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.901718 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:00:53.901753 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.901756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.901803 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.901876 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.902208 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.902996 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.904512 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.905147 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.905255 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.905302 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.905608 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.905680 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 00:00:53.906311 1686 raft_consensus.cc:504] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.906533 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.900910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:38913 (local address 127.8.204.67:55183)
0807 00:00:53.901228 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:53.901245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:53.901337 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:53.901894 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:53.901897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:53.901915 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:53.902050 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.902058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.904051 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.904055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.905193 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:53.905208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.906104 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.906108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:53.906236 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.906264 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 00:00:53.906323 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 00:00:53.906387 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":7,"thread_start_us":119,"threads_started":1}
I20260807 00:00:53.906476 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3879107b10549ec9ec83844f0366ae5" is_pre_election: true
I20260807 00:00:53.906751 1225 raft_consensus.cc:2379] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7ee41cf2916478b9b9cb3007ce9f4dc in current term 1: Already voted for candidate e3879107b10549ec9ec83844f0366ae5 in this term.
I20260807 00:00:53.907058 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.901303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:55183 (local address 127.8.204.66:38913)
0807 00:00:53.901645 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.901655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.901669 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:00:53.901715 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.901718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.901762 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.902161 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:53.902261 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.903676 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.905340 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.905933 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.906768 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.906835 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.906872 (+ 37us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.906932 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":121,"threads_started":1}
I20260807 00:00:53.907128 1407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "e3879107b10549ec9ec83844f0366ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cda1228f3a4754b15b2e38be55df39"
I20260807 00:00:53.907249 1686 leader_election.cc:288] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [CANDIDATE]: Term 1 election: Requested vote from peers b7cda1228f3a4754b15b2e38be55df39 (127.8.204.66:38913), d7ee41cf2916478b9b9cb3007ce9f4dc (127.8.204.67:44607)
I20260807 00:00:53.907277 1407 raft_consensus.cc:3037] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:53.907442 1589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "e3879107b10549ec9ec83844f0366ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc"
I20260807 00:00:53.907542 1589 raft_consensus.cc:3037] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:53.908275 1407 raft_consensus.cc:2454] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3879107b10549ec9ec83844f0366ae5 in term 1.
I20260807 00:00:53.908545 1589 raft_consensus.cc:2454] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3879107b10549ec9ec83844f0366ae5 in term 1.
I20260807 00:00:53.908613 1124 leader_election.cc:302] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7cda1228f3a4754b15b2e38be55df39, e3879107b10549ec9ec83844f0366ae5; no voters:
I20260807 00:00:53.908740 1407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cddf9b4abbfd4730b84f5b9fc33ab711" candidate_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cda1228f3a4754b15b2e38be55df39" is_pre_election: true
I20260807 00:00:53.908936 1407 raft_consensus.cc:2379] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7ee41cf2916478b9b9cb3007ce9f4dc in current term 1: Already voted for candidate e3879107b10549ec9ec83844f0366ae5 in this term.
I20260807 00:00:53.909348 1499 leader_election.cc:302] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7ee41cf2916478b9b9cb3007ce9f4dc; no voters: b7cda1228f3a4754b15b2e38be55df39, e3879107b10549ec9ec83844f0366ae5
I20260807 00:00:53.909495 1686 raft_consensus.cc:2781] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:53.909754 1686 raft_consensus.cc:686] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [term 1 LEADER]: Becoming Leader. State: Replica: e3879107b10549ec9ec83844f0366ae5, State: Running, Role: LEADER
I20260807 00:00:53.909842 1687 raft_consensus.cc:2726] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 00:00:53.910446 1686 consensus_queue.cc:237] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } }
I20260807 00:00:53.913697 1015 catalog_manager.cc:5495] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3879107b10549ec9ec83844f0366ae5 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "e3879107b10549ec9ec83844f0366ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3879107b10549ec9ec83844f0366ae5" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 41501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 } health_report { overall_health: UNKNOWN } } }
I20260807 00:00:53.956532 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:53.952854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.8.204.65:41501)
0807 00:00:53.953003 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:53.953010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:53.953030 (+ 20us) server_negotiation.cc:407] Connection header received
0807 00:00:53.953128 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:53.953131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:53.953212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:53.953333 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:53.953590 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.954094 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.955012 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:53.955239 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:53.955348 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:53.955683 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:53.956124 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:00:53.956268 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:53.956332 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 00:00:53.956386 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 00:00:53.960383 1654 tablet.cc:2369] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:53.963423 1407 raft_consensus.cc:1261] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 1 FOLLOWER]: Refusing update from remote peer e3879107b10549ec9ec83844f0366ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:53.963483 1589 raft_consensus.cc:1261] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 1 FOLLOWER]: Refusing update from remote peer e3879107b10549ec9ec83844f0366ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:53.964197 1696 consensus_queue.cc:1035] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cda1228f3a4754b15b2e38be55df39" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:53.964409 1686 consensus_queue.cc:1035] T cddf9b4abbfd4730b84f5b9fc33ab711 P e3879107b10549ec9ec83844f0366ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ee41cf2916478b9b9cb3007ce9f4dc" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:53.974733 1702 mvcc.cc:205] Tried to move back new op lower bound from 7315705257825046528 to 7315705257620226048. Current Snapshot: MvccSnapshot[applied={T|T < 7315705257825046528}]
I20260807 00:00:53.988719 1016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:53.988890 1016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:53.991493 1016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eea24315c8294831abd0eab3fd9d3a25: Sending DeleteTablet for 3 replicas of tablet cddf9b4abbfd4730b84f5b9fc33ab711
I20260807 00:00:53.992446 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:53 UTC) from {username='slave'} at 127.0.0.1:41068
I20260807 00:00:53.992724 1380 tablet_service.cc:1514] Processing DeleteTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:53 UTC) from {username='slave'} at 127.0.0.1:43464
I20260807 00:00:53.993063 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 1103
I20260807 00:00:53.993044 1562 tablet_service.cc:1514] Processing DeleteTablet for tablet cddf9b4abbfd4730b84f5b9fc33ab711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260807 00:00:53.995467 1714 tablet_replica.cc:330] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: stopping tablet replica
I20260807 00:00:53.995820 1714 raft_consensus.cc:2229] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:53.996160 1714 raft_consensus.cc:2258] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:53.996551 1715 tablet_replica.cc:330] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: stopping tablet replica
I20260807 00:00:53.996842 1715 raft_consensus.cc:2229] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:53.997129 1715 raft_consensus.cc:2258] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:54.000188 1714 ts_tablet_manager.cc:1905] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:54.000188 1715 ts_tablet_manager.cc:1905] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:54.003119 1714 ts_tablet_manager.cc:1918] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:54.003119 1715 ts_tablet_manager.cc:1918] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:54.003247 1714 log.cc:1196] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/cddf9b4abbfd4730b84f5b9fc33ab711
I20260807 00:00:54.003247 1715 log.cc:1196] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/cddf9b4abbfd4730b84f5b9fc33ab711
I20260807 00:00:54.003600 1714 ts_tablet_manager.cc:1939] T cddf9b4abbfd4730b84f5b9fc33ab711 P b7cda1228f3a4754b15b2e38be55df39: Deleting consensus metadata
I20260807 00:00:54.003597 1715 ts_tablet_manager.cc:1939] T cddf9b4abbfd4730b84f5b9fc33ab711 P d7ee41cf2916478b9b9cb3007ce9f4dc: Deleting consensus metadata
I20260807 00:00:54.005105 996 catalog_manager.cc:4841] TS d7ee41cf2916478b9b9cb3007ce9f4dc (127.8.204.67:44607): tablet cddf9b4abbfd4730b84f5b9fc33ab711 (table test-table [id=cc749a17b8754fd4accf927d5b0191f5]) successfully deleted
I20260807 00:00:54.005369 996 catalog_manager.cc:4841] TS b7cda1228f3a4754b15b2e38be55df39 (127.8.204.66:38913): tablet cddf9b4abbfd4730b84f5b9fc33ab711 (table test-table [id=cc749a17b8754fd4accf927d5b0191f5]) successfully deleted
W20260807 00:00:54.009132 1000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:41501 (error 108)
I20260807 00:00:54.010027 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 1291
I20260807 00:00:54.010408 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:54.010020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:41501 (local address 127.0.0.1:41084)
0807 00:00:54.010209 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:54.010295 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:41501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 00:00:54.016454 1000 catalog_manager.cc:4584] TS e3879107b10549ec9ec83844f0366ae5 (127.8.204.65:41501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cddf9b4abbfd4730b84f5b9fc33ab711: Network error: Client connection negotiation failed: client connection to 127.8.204.65:41501: connect: Connection refused (error 111)
I20260807 00:00:54.023826 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 1476
I20260807 00:00:54.033895 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 979
2026-08-07T00:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1300 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:00:54 dist-test-slave-39l8 krb5kdc[1719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:00:54 dist-test-slave-39l8 krb5kdc[1719](info): set up 2 sockets
Aug 07 00:00:54 dist-test-slave-39l8 krb5kdc[1719](info): commencing operation
krb5kdc: starting...
W20260807 00:00:55.249058 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:00:55 dist-test-slave-39l8 krb5kdc[1719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:00:55.379958 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45195
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:45195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:55.496884 1733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:55.497169 1733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:55.497246 1733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:55.501370 1733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:55.501436 1733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:55.501456 1733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:55.501519 1733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:55.501545 1733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:55.506934 1733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:45195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:55.508073 1733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:55.509153 1733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:55.518046 1742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:55.518070 1740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:55.518046 1738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:55.518319 1733 server_base.cc:1034] running on GCE node
I20260807 00:00:55.519093 1733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:55.520478 1733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:55.521690 1733 hybrid_clock.cc:648] HybridClock initialized: now 1786060855521674 us; error 41 us; skew 500 ppm
Aug 07 00:00:55 dist-test-slave-39l8 krb5kdc[1719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060855, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:55.524827 1733 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:00:55.525820 1733 webserver.cc:460] Webserver started at http://127.8.204.126:37591/ using document root <none> and password file <none>
I20260807 00:00:55.526463 1733 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:55.526579 1733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:55.526901 1733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:55.529258 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "6f4b40f093dc4159b5a357fc85510fa8"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "2882a42b94bf7ea64d2ca0d9fcfb36ec"
server_key_iv: "5bc2ecab6cefa9d9fd89c9189116d063"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.529917 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "6f4b40f093dc4159b5a357fc85510fa8"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "2882a42b94bf7ea64d2ca0d9fcfb36ec"
server_key_iv: "5bc2ecab6cefa9d9fd89c9189116d063"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.534127 1733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 00:00:55.536770 1752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:55.538105 1733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:55.538261 1733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "6f4b40f093dc4159b5a357fc85510fa8"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "2882a42b94bf7ea64d2ca0d9fcfb36ec"
server_key_iv: "5bc2ecab6cefa9d9fd89c9189116d063"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.538393 1733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:55.553558 1733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:55.557093 1733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:55.557313 1733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:55.564677 1733 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:45195
I20260807 00:00:55.564706 1821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:45195 every 8 connection(s)
I20260807 00:00:55.565663 1733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:55.568051 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 1733
I20260807 00:00:55.568209 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:00:55.568526 9009 external_mini_cluster.cc:1428] Setting key 02a88e01be95548c67068af3d6d11cc6
I20260807 00:00:55.568742 1823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:55.576524 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: Bootstrap starting.
Aug 07 00:00:55 dist-test-slave-39l8 krb5kdc[1719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:55.578956 1823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:55.579885 1823 log.cc:824] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:55.581887 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: No bootstrap required, opened a new log
I20260807 00:00:55.583967 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:55.570077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.8.204.126:45195)
0807 00:00:55.570542 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:55.570563 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:55.570616 (+ 53us) server_negotiation.cc:407] Connection header received
0807 00:00:55.571742 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:55.571770 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:55.572226 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:55.572623 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:55.573318 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.574827 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.575706 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.576379 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.576543 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:55.579155 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:55.579161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:55.579175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:55.579207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:55.580998 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:55.581492 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:55.581498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:55.581503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:55.581610 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:55.581945 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:55.581948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:55.581950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:55.582400 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:55.582617 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:55.582939 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 00:00:55.583242 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260807 00:00:55.585922 1823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } }
I20260807 00:00:55.586242 1823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:55.586319 1823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4b40f093dc4159b5a357fc85510fa8, State: Initialized, Role: FOLLOWER
I20260807 00:00:55.586829 1823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } }
I20260807 00:00:55.586964 1823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:55.587013 1823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:55.587097 1823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:55.587975 1823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } }
I20260807 00:00:55.588290 1823 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4b40f093dc4159b5a357fc85510fa8; no voters:
I20260807 00:00:55.588546 1823 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:00:55.588681 1828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:55.588914 1828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4b40f093dc4159b5a357fc85510fa8, State: Running, Role: LEADER
I20260807 00:00:55.589237 1828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } }
I20260807 00:00:55.589840 1823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:55.591028 1831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4b40f093dc4159b5a357fc85510fa8. Latest consensus state: current_term: 1 leader_uuid: "6f4b40f093dc4159b5a357fc85510fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } } }
I20260807 00:00:55.591166 1831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:55.591483 1829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4b40f093dc4159b5a357fc85510fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4b40f093dc4159b5a357fc85510fa8" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 45195 } } }
I20260807 00:00:55.591594 1829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:55.592271 1842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:55.595088 1842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:55.601620 1842 catalog_manager.cc:1347] Generated new cluster ID: dd1d53a9575c4288825ace2be781beb3
I20260807 00:00:55.601720 1842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:55.629007 1842 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:00:55.630301 1842 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:55.641729 1842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: Generated new TSK 0
I20260807 00:00:55.642616 1842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260807 00:00:55.677625 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:55.797034 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:55.797307 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:55.797351 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:55.801306 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:55.801366 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:55.801424 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:55.806807 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:55.807842 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:55.808925 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:55.818948 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:55.818954 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:55.818954 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:55.819203 1862 server_base.cc:1034] running on GCE node
I20260807 00:00:55.819796 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:55.820513 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:55.821686 1862 hybrid_clock.cc:648] HybridClock initialized: now 1786060855821662 us; error 36 us; skew 500 ppm
Aug 07 00:00:55 dist-test-slave-39l8 krb5kdc[1719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060855, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:55.824649 1862 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260807 00:00:55.825618 1862 webserver.cc:460] Webserver started at http://127.8.204.65:46871/ using document root <none> and password file <none>
I20260807 00:00:55.826221 1862 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:55.826316 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:55.826565 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:55.828747 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "847a00a284d048d4b6fb3a0c27bd8b0e"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "54ed09bcb029beabc291bc18b91f18cf"
server_key_iv: "ff8af72ac21c1926becf44a4ede7cdb2"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.829398 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "847a00a284d048d4b6fb3a0c27bd8b0e"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "54ed09bcb029beabc291bc18b91f18cf"
server_key_iv: "ff8af72ac21c1926becf44a4ede7cdb2"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.833451 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 00:00:55.836074 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:55.837324 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:00:55.837469 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "847a00a284d048d4b6fb3a0c27bd8b0e"
format_stamp: "Formatted at 2026-08-07 00:00:55 on dist-test-slave-39l8"
server_key: "54ed09bcb029beabc291bc18b91f18cf"
server_key_iv: "ff8af72ac21c1926becf44a4ede7cdb2"
server_key_version: "encryptionkey@0"
I20260807 00:00:55.837590 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:55.848031 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:55.851413 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:55.851624 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:55.852259 1862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:55.853135 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:55.853206 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:55.853282 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:55.853327 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:55.862517 1862 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:45839
I20260807 00:00:55.862596 2045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:45839 every 8 connection(s)
I20260807 00:00:55.863498 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:55.865562 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 1862
I20260807 00:00:55.865711 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:00:55.866047 9009 external_mini_cluster.cc:1428] Setting key 7ec723969a039481e8bb9632933532e5
Aug 07 00:00:55 dist-test-slave-39l8 krb5kdc[1719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060855, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
I20260807 00:00:55.880896 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:55.865612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41721 (local address 127.8.204.126:45195)
0807 00:00:55.865797 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:55.865802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:55.866828 (+ 1026us) server_negotiation.cc:407] Connection header received
0807 00:00:55.868763 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:55.868766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:55.868811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:55.868880 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:55.870609 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.871517 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.873534 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.873782 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.873874 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:55.877139 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:55.877142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:55.877144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:55.877170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:55.878757 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:55.879332 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:55.879335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:55.879336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:55.879417 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:55.879774 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:55.879776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:55.879777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:55.879959 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:55.880038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:55.880535 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 00:00:55.880762 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260807 00:00:55.881783 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:55.866025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45195 (local address 127.8.204.65:41721)
0807 00:00:55.866615 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:55.866662 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:55.868517 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:55.869083 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:55.869097 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:55.869703 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:55.870435 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:55.870452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.871671 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.871674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:55.873382 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:55.873398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:55.873969 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:55.873975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:55.874225 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:55.874996 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:55.875018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:55.876930 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:55.878889 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:55.878898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:55.878917 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:55.879236 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:55.879514 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:55.879516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:55.879517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:55.879699 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:55.880064 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:55.880069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:55.880315 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 00:00:55.880484 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 00:00:55.880776 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":126,"threads_started":1}
I20260807 00:00:55.883265 2047 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45195
I20260807 00:00:55.883536 2047 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:55.884053 2047 heartbeater.cc:511] Master 127.8.204.126:45195 requested a full tablet report, sending...
I20260807 00:00:55.885820 1776 ts_manager.cc:194] Registered new tserver with Master: 847a00a284d048d4b6fb3a0c27bd8b0e (127.8.204.65:45839)
I20260807 00:00:55.887667 1776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:41721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260807 00:00:55.897775 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:56.014084 2056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:56.014428 2056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:56.014536 2056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:56.019739 2056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:56.019830 2056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:56.019929 2056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:56.026500 2056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:56.027714 2056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:56.028851 2056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 00:00:56.038913 2056 server_base.cc:1034] running on GCE node
W20260807 00:00:56.038904 2062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.039111 2061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.038910 2064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:56.039707 2056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:56.040371 2056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:56.041577 2056 hybrid_clock.cc:648] HybridClock initialized: now 1786060856041557 us; error 39 us; skew 500 ppm
Aug 07 00:00:56 dist-test-slave-39l8 krb5kdc[1719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060856, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:56.044674 2056 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260807 00:00:56.045590 2056 webserver.cc:460] Webserver started at http://127.8.204.66:46627/ using document root <none> and password file <none>
I20260807 00:00:56.046190 2056 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:56.046284 2056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:56.046520 2056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:56.048671 2056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "6c430d07555544a886c0873481499aac"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "ca0b7bbd246824380b67e367a6f9f7be"
server_key_iv: "4373b9d02db90f049816cdd33c2efd59"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.049276 2056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "6c430d07555544a886c0873481499aac"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "ca0b7bbd246824380b67e367a6f9f7be"
server_key_iv: "4373b9d02db90f049816cdd33c2efd59"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.053290 2056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 00:00:56.056005 2077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.057317 2056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:00:56.057463 2056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "6c430d07555544a886c0873481499aac"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "ca0b7bbd246824380b67e367a6f9f7be"
server_key_iv: "4373b9d02db90f049816cdd33c2efd59"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.057587 2056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:56.067574 2056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:56.070641 2056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:56.070907 2056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:56.071527 2056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:56.072413 2056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:56.072485 2056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.072558 2056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:56.072610 2056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.082059 2056 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:42285
I20260807 00:00:56.082078 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:42285 every 8 connection(s)
I20260807 00:00:56.083088 2056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:56.085356 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 2056
I20260807 00:00:56.085515 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:00:56.085870 9009 external_mini_cluster.cc:1428] Setting key e02151970e420e12214dc94d8cd3dd94
Aug 07 00:00:56 dist-test-slave-39l8 krb5kdc[1719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060856, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260807 00:00:56.100278 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.085163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:44363 (local address 127.8.204.126:45195)
0807 00:00:56.085332 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.085337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.086317 (+ 980us) server_negotiation.cc:407] Connection header received
0807 00:00:56.088149 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.088152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.088201 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.088272 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:56.090129 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.091059 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.092922 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.093140 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.093227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.096268 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:56.096274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:56.096280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:56.096318 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:56.098072 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.098664 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.098671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.098676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.098800 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.099158 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.099161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.099163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.099421 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:56.099511 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.100030 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.100148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260807 00:00:56.101406 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.085524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45195 (local address 127.8.204.66:44363)
0807 00:00:56.086127 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.086191 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.087902 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.088459 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.088471 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.089034 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:56.089909 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.089938 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.091215 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.091219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.092775 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.092791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.093319 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.093328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.093598 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.094263 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:56.094291 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:56.096078 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:56.098207 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:56.098212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:56.098226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:56.098554 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:56.098887 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:56.098889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:56.098890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:56.099037 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:56.099527 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:56.099533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:56.099789 (+ 256us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.099962 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.100382 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":119,"threads_started":1}
I20260807 00:00:56.102694 2247 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45195
I20260807 00:00:56.103008 2247 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:56.103613 2247 heartbeater.cc:511] Master 127.8.204.126:45195 requested a full tablet report, sending...
I20260807 00:00:56.104941 1776 ts_manager.cc:194] Registered new tserver with Master: 6c430d07555544a886c0873481499aac (127.8.204.66:42285)
I20260807 00:00:56.105553 1776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:44363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260807 00:00:56.118306 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:56.235069 2252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:56.235479 2252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:56.235548 2252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:56.239657 2252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:56.239732 2252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:56.239790 2252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:56.245412 2252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:56.246485 2252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:56.247578 2252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:56.257477 2262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.257477 2264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.257498 2261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:56.257722 2252 server_base.cc:1034] running on GCE node
I20260807 00:00:56.258262 2252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:56.258975 2252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:56.260144 2252 hybrid_clock.cc:648] HybridClock initialized: now 1786060856260111 us; error 45 us; skew 500 ppm
Aug 07 00:00:56 dist-test-slave-39l8 krb5kdc[1719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060856, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:56.263020 2252 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260807 00:00:56.264012 2252 webserver.cc:460] Webserver started at http://127.8.204.67:39149/ using document root <none> and password file <none>
I20260807 00:00:56.264612 2252 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:56.264678 2252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:56.264937 2252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:56.267107 2252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "461228faa95f4d21868b742ce7cc1268"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "aa0557bf83d3abd8f7528b4011055ad3"
server_key_iv: "f861d81e4856ce380bc450294d35debe"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.267704 2252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "461228faa95f4d21868b742ce7cc1268"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "aa0557bf83d3abd8f7528b4011055ad3"
server_key_iv: "f861d81e4856ce380bc450294d35debe"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.271584 2252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:00:56.274209 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.275560 2252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:00:56.275712 2252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "461228faa95f4d21868b742ce7cc1268"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "aa0557bf83d3abd8f7528b4011055ad3"
server_key_iv: "f861d81e4856ce380bc450294d35debe"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.275835 2252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:56.313131 2252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:56.316334 2252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:56.316552 2252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:56.317207 2252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:56.318096 2252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:56.318181 2252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.318270 2252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:56.318321 2252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.327339 2252 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:46155
I20260807 00:00:56.327415 2443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:46155 every 8 connection(s)
I20260807 00:00:56.328298 2252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:00:56.337088 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 2252
I20260807 00:00:56.337272 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:00:56.337577 9009 external_mini_cluster.cc:1428] Setting key 802f7d95a9f981f2dd78a16a3b2f70f9
Aug 07 00:00:56 dist-test-slave-39l8 krb5kdc[1719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060856, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:56.342528 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.330171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:40585 (local address 127.8.204.126:45195)
0807 00:00:56.330313 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.330317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.331247 (+ 930us) server_negotiation.cc:407] Connection header received
0807 00:00:56.332679 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.332682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.332732 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.332809 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:56.333989 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.334560 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.335720 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.335935 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.336020 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.338828 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:56.338835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:56.338841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:56.338876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:56.340389 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.340929 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.340932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.340933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.341013 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.341353 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.341355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.341357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.341548 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:56.341626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.342231 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.342405 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:56.343505 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.330541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:45195 (local address 127.8.204.67:40585)
0807 00:00:56.331066 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.331109 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.332473 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.332975 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.332981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.333341 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:56.333822 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.333836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.334673 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.334676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.335597 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.335607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.336074 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.336076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.336223 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.336866 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:56.336882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:56.338593 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:56.340502 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:56.340507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:56.340520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:56.340807 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:56.341096 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:56.341099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:56.341100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:56.341270 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:56.341661 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:56.341666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:56.341919 (+ 253us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.342153 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.342493 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":185,"threads_started":1}
I20260807 00:00:56.344646 2444 heartbeater.cc:347] Connected to a master server at 127.8.204.126:45195
I20260807 00:00:56.344872 2444 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:56.345351 2444 heartbeater.cc:511] Master 127.8.204.126:45195 requested a full tablet report, sending...
I20260807 00:00:56.346350 1776 ts_manager.cc:194] Registered new tserver with Master: 461228faa95f4d21868b742ce7cc1268 (127.8.204.67:46155)
I20260807 00:00:56.346969 1776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:40585
I20260807 00:00:56.351904 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:00:56.361361 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.353934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.8.204.126:45195)
0807 00:00:56.354069 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.354073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.354262 (+ 189us) server_negotiation.cc:407] Connection header received
0807 00:00:56.354352 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.354354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.354392 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.354462 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:56.355261 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.355768 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.356546 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.356753 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.356829 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.358298 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:56.358301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:56.358303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:56.358326 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:56.359626 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.360088 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.360090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.360092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.360179 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:56.360458 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:56.360460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:56.360462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:56.360657 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:56.360831 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.361142 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.361234 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 00:00:56.364365 1776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:00:56.367455 1776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:00:56.383005 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.376423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45839 (local address 127.0.0.1:51290)
0807 00:00:56.376793 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.376840 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.376964 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.378483 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.378489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.378511 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:56.378639 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.378644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380248 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.380252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.381485 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.381495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.382471 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.382474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.382557 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.382592 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.382685 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.382856 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260807 00:00:56.383056 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.377276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.8.204.65:45839)
0807 00:00:56.377984 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.377999 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.378022 (+ 23us) server_negotiation.cc:407] Connection header received
0807 00:00:56.378101 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.378107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.378316 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.378480 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:56.378809 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380070 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.381626 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.382345 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.382455 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.382602 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.382770 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.382878 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":73,"threads_started":1}
I20260807 00:00:56.383819 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.377249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.8.204.67:46155)
0807 00:00:56.377984 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.377997 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.378022 (+ 25us) server_negotiation.cc:407] Connection header received
0807 00:00:56.378101 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.378107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.378316 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.378480 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:56.378991 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380107 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.382053 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.382905 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.383014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.383149 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.383548 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.383671 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":99,"threads_started":1}
I20260807 00:00:56.383445 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.376721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:46155 (local address 127.0.0.1:39980)
0807 00:00:56.377653 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.377668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.377748 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.378671 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.378675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.378752 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:56.378854 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.378861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380462 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.380466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.381916 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.381928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.383035 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.383041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.383138 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.383176 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.383257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.383316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":27,"thread_start_us":80,"threads_started":1}
I20260807 00:00:56.384344 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.376455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:42285 (local address 127.0.0.1:42318)
0807 00:00:56.377621 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.377635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.377736 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.379244 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.379246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.379265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:56.379386 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.379394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380593 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.380597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.381917 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.381928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.383900 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.383904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.383989 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.384009 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.384102 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.384179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":86,"thread_start_us":119,"threads_started":1}
I20260807 00:00:56.384756 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.377369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.8.204.66:42285)
0807 00:00:56.378807 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.378828 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.378852 (+ 24us) server_negotiation.cc:407] Connection header received
0807 00:00:56.378929 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.378937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.379094 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.379261 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:56.379482 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.380375 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.383217 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.383768 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.383899 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.384037 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.384499 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.384610 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":133,"threads_started":1}
I20260807 00:00:56.387133 2352 tablet_service.cc:1467] Processing CreateTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 (DEFAULT_TABLE table=test-table [id=ba996f888f2f4276b370b0b33496e627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:56.387208 1949 tablet_service.cc:1467] Processing CreateTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 (DEFAULT_TABLE table=test-table [id=ba996f888f2f4276b370b0b33496e627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:56.387552 2142 tablet_service.cc:1467] Processing CreateTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 (DEFAULT_TABLE table=test-table [id=ba996f888f2f4276b370b0b33496e627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:56.388340 1949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fef70bc3e794f4f8ae88b89397cd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:56.388361 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fef70bc3e794f4f8ae88b89397cd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:56.388413 2142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fef70bc3e794f4f8ae88b89397cd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:56.395076 2473 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Bootstrap starting.
I20260807 00:00:56.395927 2477 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: Bootstrap starting.
I20260807 00:00:56.396203 2475 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Bootstrap starting.
I20260807 00:00:56.397774 2473 tablet_bootstrap.cc:654] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:56.398139 2477 tablet_bootstrap.cc:654] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:56.398628 2475 tablet_bootstrap.cc:654] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:56.399161 2477 log.cc:824] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:56.399482 2475 log.cc:824] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:56.399761 2473 log.cc:824] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:56.401399 2475 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: No bootstrap required, opened a new log
I20260807 00:00:56.401424 2477 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: No bootstrap required, opened a new log
I20260807 00:00:56.401621 2475 ts_tablet_manager.cc:1397] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:56.401620 2477 ts_tablet_manager.cc:1397] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:56.401718 2473 tablet_bootstrap.cc:492] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: No bootstrap required, opened a new log
I20260807 00:00:56.401889 2473 ts_tablet_manager.cc:1397] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 00:00:56.404621 2475 raft_consensus.cc:348] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.404881 2475 raft_consensus.cc:374] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:56.404955 2475 raft_consensus.cc:729] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c430d07555544a886c0873481499aac, State: Initialized, Role: FOLLOWER
I20260807 00:00:56.405441 2475 consensus_queue.cc:260] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.405794 2473 raft_consensus.cc:348] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.406085 2473 raft_consensus.cc:374] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:56.406174 2473 raft_consensus.cc:729] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461228faa95f4d21868b742ce7cc1268, State: Initialized, Role: FOLLOWER
I20260807 00:00:56.406069 2477 raft_consensus.cc:348] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.406324 2477 raft_consensus.cc:374] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:56.406383 2477 raft_consensus.cc:729] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847a00a284d048d4b6fb3a0c27bd8b0e, State: Initialized, Role: FOLLOWER
I20260807 00:00:56.406576 2475 ts_tablet_manager.cc:1428] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:00:56.406653 2473 consensus_queue.cc:260] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.406757 2477 consensus_queue.cc:260] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.407013 2247 heartbeater.cc:503] Master 127.8.204.126:45195 was elected leader, sending a full tablet report...
I20260807 00:00:56.407621 2473 ts_tablet_manager.cc:1428] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 00:00:56.407621 2477 ts_tablet_manager.cc:1428] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 00:00:56.407945 2047 heartbeater.cc:503] Master 127.8.204.126:45195 was elected leader, sending a full tablet report...
I20260807 00:00:56.408089 2444 heartbeater.cc:503] Master 127.8.204.126:45195 was elected leader, sending a full tablet report...
I20260807 00:00:56.532979 2484 raft_consensus.cc:482] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:56.533195 2484 raft_consensus.cc:504] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.534358 2484 leader_election.cc:288] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 847a00a284d048d4b6fb3a0c27bd8b0e (127.8.204.65:45839), 461228faa95f4d21868b742ce7cc1268 (127.8.204.67:46155)
I20260807 00:00:56.539034 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.534592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45839 (local address 127.8.204.66:40373)
0807 00:00:56.534777 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.534794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.534898 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.535196 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.535199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.535222 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:56.535347 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.535355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.536372 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.536375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.537652 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.537664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.538524 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.538528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.538610 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.538631 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.538761 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.538864 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":29}
I20260807 00:00:56.539026 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.534743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:40373 (local address 127.8.204.65:45839)
0807 00:00:56.534912 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.534918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.534937 (+ 19us) server_negotiation.cc:407] Connection header received
0807 00:00:56.535004 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.535010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.535070 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.535164 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:56.535480 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.536242 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.537785 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.538392 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.538519 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.538577 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.538795 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.538880 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 00:00:56.539225 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.534777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:45367 (local address 127.8.204.67:46155)
0807 00:00:56.534897 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.534902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.535460 (+ 558us) server_negotiation.cc:407] Connection header received
0807 00:00:56.535563 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.535566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.535613 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.535698 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:56.535936 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.536563 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.537972 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.538595 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.538739 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.538799 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.539002 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.539096 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 00:00:56.539232 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.534596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:46155 (local address 127.8.204.66:45367)
0807 00:00:56.535347 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.535361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:56.535460 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:56.535716 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:56.535718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:56.535734 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:56.535824 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.535828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.536681 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.536684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.537850 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:56.537858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.538745 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.538751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:56.538876 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.538896 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 00:00:56.538970 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 00:00:56.539036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":269,"thread_start_us":184,"threads_started":1}
I20260807 00:00:56.539919 1973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fef70bc3e794f4f8ae88b89397cd317" candidate_uuid: "6c430d07555544a886c0873481499aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" is_pre_election: true
I20260807 00:00:56.539956 2380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fef70bc3e794f4f8ae88b89397cd317" candidate_uuid: "6c430d07555544a886c0873481499aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461228faa95f4d21868b742ce7cc1268" is_pre_election: true
I20260807 00:00:56.540290 2380 raft_consensus.cc:2454] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c430d07555544a886c0873481499aac in term 0.
I20260807 00:00:56.540292 1973 raft_consensus.cc:2454] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c430d07555544a886c0873481499aac in term 0.
I20260807 00:00:56.540905 2081 leader_election.cc:302] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 461228faa95f4d21868b742ce7cc1268, 6c430d07555544a886c0873481499aac, 847a00a284d048d4b6fb3a0c27bd8b0e; no voters:
I20260807 00:00:56.541227 2484 raft_consensus.cc:2781] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:00:56.541319 2484 raft_consensus.cc:482] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:00:56.541373 2484 raft_consensus.cc:3037] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:56.542598 2484 raft_consensus.cc:504] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.542984 2484 leader_election.cc:288] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [CANDIDATE]: Term 1 election: Requested vote from peers 847a00a284d048d4b6fb3a0c27bd8b0e (127.8.204.65:45839), 461228faa95f4d21868b742ce7cc1268 (127.8.204.67:46155)
I20260807 00:00:56.543362 1973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fef70bc3e794f4f8ae88b89397cd317" candidate_uuid: "6c430d07555544a886c0873481499aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e"
I20260807 00:00:56.543437 2380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fef70bc3e794f4f8ae88b89397cd317" candidate_uuid: "6c430d07555544a886c0873481499aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461228faa95f4d21868b742ce7cc1268"
I20260807 00:00:56.543497 1973 raft_consensus.cc:3037] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:56.543556 2380 raft_consensus.cc:3037] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:56.544860 2380 raft_consensus.cc:2454] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c430d07555544a886c0873481499aac in term 1.
I20260807 00:00:56.544860 1973 raft_consensus.cc:2454] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c430d07555544a886c0873481499aac in term 1.
I20260807 00:00:56.545198 2081 leader_election.cc:302] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c430d07555544a886c0873481499aac, 847a00a284d048d4b6fb3a0c27bd8b0e; no voters:
I20260807 00:00:56.545449 2484 raft_consensus.cc:2781] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:56.545652 2484 raft_consensus.cc:686] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 1 LEADER]: Becoming Leader. State: Replica: 6c430d07555544a886c0873481499aac, State: Running, Role: LEADER
I20260807 00:00:56.546020 2484 consensus_queue.cc:237] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } }
I20260807 00:00:56.549460 1775 catalog_manager.cc:5495] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c430d07555544a886c0873481499aac (127.8.204.66). New cstate: current_term: 1 leader_uuid: "6c430d07555544a886c0873481499aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c430d07555544a886c0873481499aac" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 } health_report { overall_health: UNKNOWN } } }
I20260807 00:00:56.576413 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.572210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.8.204.66:42285)
0807 00:00:56.572357 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.572362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.572378 (+ 16us) server_negotiation.cc:407] Connection header received
0807 00:00:56.572469 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.572481 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.572556 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.572681 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:56.572934 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.573729 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.574638 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.575002 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.575105 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.575421 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:56.575927 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:00:56.576069 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.576220 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.576265 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 00:00:56.579425 2446 tablet.cc:2369] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:56.583000 1973 raft_consensus.cc:1261] T 8fef70bc3e794f4f8ae88b89397cd317 P 847a00a284d048d4b6fb3a0c27bd8b0e [term 1 FOLLOWER]: Refusing update from remote peer 6c430d07555544a886c0873481499aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:56.583000 2380 raft_consensus.cc:1261] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 1 FOLLOWER]: Refusing update from remote peer 6c430d07555544a886c0873481499aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:56.583787 2484 consensus_queue.cc:1035] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "461228faa95f4d21868b742ce7cc1268" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:56.584015 2488 consensus_queue.cc:1035] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "847a00a284d048d4b6fb3a0c27bd8b0e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 00:00:56.584326 2248 tablet.cc:2369] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:56.590663 2496 mvcc.cc:205] Tried to move back new op lower bound from 7315705268556111872 to 7315705268415574016. Current Snapshot: MvccSnapshot[applied={T|T < 7315705268556111872}]
I20260807 00:00:56.594990 2497 mvcc.cc:205] Tried to move back new op lower bound from 7315705268556111872 to 7315705268415574016. Current Snapshot: MvccSnapshot[applied={T|T < 7315705268556111872}]
I20260807 00:00:56.600418 1775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:56.600610 1775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:56.603272 1775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f4b40f093dc4159b5a357fc85510fa8: Sending DeleteTablet for 3 replicas of tablet 8fef70bc3e794f4f8ae88b89397cd317
I20260807 00:00:56.604333 1949 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:51290
I20260807 00:00:56.604403 2142 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:42318
I20260807 00:00:56.604956 2352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fef70bc3e794f4f8ae88b89397cd317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:56 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:39980
I20260807 00:00:56.605110 2511 tablet_replica.cc:330] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: stopping tablet replica
I20260807 00:00:56.605171 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 1862
I20260807 00:00:56.605935 2511 raft_consensus.cc:2229] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 1 LEADER]: Raft consensus shutting down.
I20260807 00:00:56.606446 2511 raft_consensus.cc:2258] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:56.607668 2514 tablet_replica.cc:330] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: stopping tablet replica
I20260807 00:00:56.608006 2514 raft_consensus.cc:2229] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:56.608409 2514 raft_consensus.cc:2258] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:56.611737 2511 ts_tablet_manager.cc:1905] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:56.611738 2514 ts_tablet_manager.cc:1905] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:56.614804 2511 ts_tablet_manager.cc:1918] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:56.614918 2511 log.cc:1196] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/8fef70bc3e794f4f8ae88b89397cd317
I20260807 00:00:56.615265 2511 ts_tablet_manager.cc:1939] T 8fef70bc3e794f4f8ae88b89397cd317 P 6c430d07555544a886c0873481499aac: Deleting consensus metadata
I20260807 00:00:56.615278 2514 ts_tablet_manager.cc:1918] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:56.615381 2514 log.cc:1196] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/8fef70bc3e794f4f8ae88b89397cd317
I20260807 00:00:56.615744 2514 ts_tablet_manager.cc:1939] T 8fef70bc3e794f4f8ae88b89397cd317 P 461228faa95f4d21868b742ce7cc1268: Deleting consensus metadata
I20260807 00:00:56.616614 1753 catalog_manager.cc:4841] TS 6c430d07555544a886c0873481499aac (127.8.204.66:42285): tablet 8fef70bc3e794f4f8ae88b89397cd317 (table test-table [id=ba996f888f2f4276b370b0b33496e627]) successfully deleted
I20260807 00:00:56.617020 1755 catalog_manager.cc:4841] TS 461228faa95f4d21868b742ce7cc1268 (127.8.204.67:46155): tablet 8fef70bc3e794f4f8ae88b89397cd317 (table test-table [id=ba996f888f2f4276b370b0b33496e627]) successfully deleted
W20260807 00:00:56.621755 1757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:45839 (error 108)
I20260807 00:00:56.621994 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 2056
I20260807 00:00:56.629691 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.629164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:45839 (local address 127.0.0.1:51302)
0807 00:00:56.629455 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:56.629562 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260807 00:00:56.629951 1757 catalog_manager.cc:4584] TS 847a00a284d048d4b6fb3a0c27bd8b0e (127.8.204.65:45839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fef70bc3e794f4f8ae88b89397cd317: Network error: Client connection negotiation failed: client connection to 127.8.204.65:45839: connect: Connection refused (error 111)
I20260807 00:00:56.636782 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 2252
I20260807 00:00:56.651470 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 1733
2026-08-07T00:00:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2615 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T00:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:00:56Z Disabled control of system clock
I20260807 00:00:56.689862 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38929
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:38929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:00:56.808499 2519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:56.808813 2519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:56.808851 2519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:56.812884 2519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:56.812950 2519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:56.812966 2519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:56.812984 2519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:56.812999 2519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:56.817951 2519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:38929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:56.819130 2519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:56.820535 2519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:56.829260 2527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.829275 2525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:56.829289 2531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:56.829552 2519 server_base.cc:1034] running on GCE node
I20260807 00:00:56.830257 2519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:56.831403 2519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:56.832573 2519 hybrid_clock.cc:648] HybridClock initialized: now 1786060856832541 us; error 44 us; skew 500 ppm
I20260807 00:00:56.834411 2519 webserver.cc:460] Webserver started at http://127.8.204.126:43177/ using document root <none> and password file <none>
I20260807 00:00:56.835059 2519 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:56.835129 2519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:56.835306 2519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:56.837360 2519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "1c2b9e1c31ee44da95449bbba1effbda"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "c6e0d22989e1af14dc58a8fb16252289"
server_key_iv: "e4d6c63bc9d52a2ba094a532d85a8159"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.837909 2519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "1c2b9e1c31ee44da95449bbba1effbda"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "c6e0d22989e1af14dc58a8fb16252289"
server_key_iv: "e4d6c63bc9d52a2ba094a532d85a8159"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.841773 2519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 00:00:56.844540 2537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:56.845844 2519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 00:00:56.846006 2519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "1c2b9e1c31ee44da95449bbba1effbda"
format_stamp: "Formatted at 2026-08-07 00:00:56 on dist-test-slave-39l8"
server_key: "c6e0d22989e1af14dc58a8fb16252289"
server_key_iv: "e4d6c63bc9d52a2ba094a532d85a8159"
server_key_version: "encryptionkey@0"
I20260807 00:00:56.846131 2519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:56.855160 2519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:56.855865 2519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:56.856061 2519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:56.863344 2519 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:38929
I20260807 00:00:56.863361 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:38929 every 8 connection(s)
I20260807 00:00:56.864301 2519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:56.866811 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 2519
I20260807 00:00:56.866951 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:00:56.867252 9009 external_mini_cluster.cc:1428] Setting key eccaf803a3cb853ef67282d13c0f08a3
I20260807 00:00:56.867583 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:56.874246 2614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: Bootstrap starting.
I20260807 00:00:56.876844 2614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:56.876947 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:56.868693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.8.204.126:38929)
0807 00:00:56.869356 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:56.869371 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:56.869406 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:00:56.870370 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:56.870391 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:56.870863 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:56.871218 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:56.871648 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.872615 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.873520 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:56.873988 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:56.874163 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:56.875117 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:56.875123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:56.875135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:56.875532 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:56.875590 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:56.875611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:56.875727 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:56.875868 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 00:00:56.876142 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":165,"threads_started":1}
I20260807 00:00:56.877840 2614 log.cc:824] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:56.880002 2614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: No bootstrap required, opened a new log
I20260807 00:00:56.883715 2614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } }
I20260807 00:00:56.883948 2614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:56.883984 2614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2b9e1c31ee44da95449bbba1effbda, State: Initialized, Role: FOLLOWER
I20260807 00:00:56.884407 2614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } }
I20260807 00:00:56.884529 2614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:56.884603 2614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:56.884711 2614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:56.885643 2614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } }
I20260807 00:00:56.886034 2614 leader_election.cc:302] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c2b9e1c31ee44da95449bbba1effbda; no voters:
I20260807 00:00:56.886404 2614 leader_election.cc:288] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:00:56.886493 2625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:56.886687 2625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [term 1 LEADER]: Becoming Leader. State: Replica: 1c2b9e1c31ee44da95449bbba1effbda, State: Running, Role: LEADER
I20260807 00:00:56.887023 2625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } }
I20260807 00:00:56.887794 2614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:56.888860 2630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c2b9e1c31ee44da95449bbba1effbda. Latest consensus state: current_term: 1 leader_uuid: "1c2b9e1c31ee44da95449bbba1effbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } } }
I20260807 00:00:56.888993 2630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:56.889266 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c2b9e1c31ee44da95449bbba1effbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2b9e1c31ee44da95449bbba1effbda" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 38929 } } }
I20260807 00:00:56.889364 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:56.890318 2640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:56.893424 2640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:56.899459 2640 catalog_manager.cc:1347] Generated new cluster ID: d5c74d9d28fc403d9b85cefd26a65d97
I20260807 00:00:56.899549 2640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:56.929421 2640 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:00:56.930601 2640 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:56.938799 2640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: Generated new TSK 0
I20260807 00:00:56.939661 2640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:00:56.955796 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--builtin_ntp_servers=127.8.204.84:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:00:57.076262 2653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:57.076614 2653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:57.076700 2653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:57.080929 2653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:57.081065 2653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:57.081158 2653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:57.086079 2653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:57.087332 2653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:57.088909 2653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:57.099217 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:57.099237 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:57.099241 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:57.099705 2653 server_base.cc:1034] running on GCE node
I20260807 00:00:57.100095 2653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:57.100714 2653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:57.101884 2653 hybrid_clock.cc:648] HybridClock initialized: now 1786060857101854 us; error 41 us; skew 500 ppm
I20260807 00:00:57.103793 2653 webserver.cc:460] Webserver started at http://127.8.204.65:42891/ using document root <none> and password file <none>
I20260807 00:00:57.104384 2653 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:57.104446 2653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:57.104624 2653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:57.106837 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "dda31eae3beb41d58c67ac8225bfe61e"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "1e6db34b7c1f71ff0376d5ba51439b06"
server_key_iv: "3d845c225d4e201340fc5d8c2d57bcdc"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.107419 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "dda31eae3beb41d58c67ac8225bfe61e"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "1e6db34b7c1f71ff0376d5ba51439b06"
server_key_iv: "3d845c225d4e201340fc5d8c2d57bcdc"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.111274 2653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 00:00:57.113814 2673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.115109 2653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:57.115249 2653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "dda31eae3beb41d58c67ac8225bfe61e"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "1e6db34b7c1f71ff0376d5ba51439b06"
server_key_iv: "3d845c225d4e201340fc5d8c2d57bcdc"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.115365 2653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:57.133543 2653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:57.134333 2653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:57.134549 2653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:57.135234 2653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:57.136278 2653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:57.136353 2653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.136430 2653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:57.136473 2653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.145834 2653 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:35481
I20260807 00:00:57.145862 2823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:35481 every 8 connection(s)
I20260807 00:00:57.146970 2653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:57.154194 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 2653
I20260807 00:00:57.154371 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:00:57.154682 9009 external_mini_cluster.cc:1428] Setting key 3447996156355bd5295cff907b69b12c
I20260807 00:00:57.156888 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--builtin_ntp_servers=127.8.204.84:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:00:57.156903 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.148969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:48605 (local address 127.8.204.126:38929)
0807 00:00:57.149142 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.149148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.150169 (+ 1021us) server_negotiation.cc:407] Connection header received
0807 00:00:57.151774 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.151777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.151847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.151925 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:57.152840 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.153377 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.154593 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.154943 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.155061 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.156362 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:57.156365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:57.156368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:57.156381 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:57.156405 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:57.156419 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:57.156499 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.156663 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.156741 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 00:00:57.157763 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.149329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38929 (local address 127.8.204.65:48605)
0807 00:00:57.149998 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.150040 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.151540 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.152112 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.152119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.152208 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:57.152692 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.152705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.153513 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.153522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.154460 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.154475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.155111 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.155116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.155329 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.156013 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:57.156014 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:57.156097 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:57.156206 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:57.156500 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:57.156513 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:57.156521 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.156663 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.156957 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260807 00:00:57.159168 2824 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38929
I20260807 00:00:57.159448 2824 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:57.160002 2824 heartbeater.cc:511] Master 127.8.204.126:38929 requested a full tablet report, sending...
I20260807 00:00:57.161801 2561 ts_manager.cc:194] Registered new tserver with Master: dda31eae3beb41d58c67ac8225bfe61e (127.8.204.65:35481)
I20260807 00:00:57.163725 2561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.65:48605
W20260807 00:00:57.276942 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:57.277318 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:57.277391 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:57.281631 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:57.281692 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:57.281747 2827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:00:57.286589 2827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:57.287851 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:57.289398 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:57.299386 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:57.299423 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:57.299391 2834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:57.299790 2827 server_base.cc:1034] running on GCE node
I20260807 00:00:57.300218 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:57.300889 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:57.302079 2827 hybrid_clock.cc:648] HybridClock initialized: now 1786060857302076 us; error 45 us; skew 500 ppm
I20260807 00:00:57.303992 2827 webserver.cc:460] Webserver started at http://127.8.204.66:39905/ using document root <none> and password file <none>
I20260807 00:00:57.304598 2827 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:57.304703 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:57.304948 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:57.307140 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "7f4b41f34b614dde98ba25223d967bef"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "144faf720cd0f4395f464baec8071313"
server_key_iv: "30f2dd30288e13cad1fcd18980d3cca2"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.307741 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "7f4b41f34b614dde98ba25223d967bef"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "144faf720cd0f4395f464baec8071313"
server_key_iv: "30f2dd30288e13cad1fcd18980d3cca2"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.311659 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:00:57.314129 2846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.315362 2827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:57.315514 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "7f4b41f34b614dde98ba25223d967bef"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "144faf720cd0f4395f464baec8071313"
server_key_iv: "30f2dd30288e13cad1fcd18980d3cca2"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.315636 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:57.328295 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:57.328996 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:57.329200 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:57.329807 2827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:57.330845 2827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:57.330914 2827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.330988 2827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:57.331029 2827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.339963 2827 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:43315
I20260807 00:00:57.340003 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:43315 every 8 connection(s)
I20260807 00:00:57.340893 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:00:57.345695 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 2827
I20260807 00:00:57.345830 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:00:57.346170 9009 external_mini_cluster.cc:1428] Setting key 3e65855826fade13756c6184e22d3939
I20260807 00:00:57.348171 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--builtin_ntp_servers=127.8.204.84:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:00:57.351003 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.342779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48415 (local address 127.8.204.126:38929)
0807 00:00:57.342933 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.342937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.343980 (+ 1043us) server_negotiation.cc:407] Connection header received
0807 00:00:57.345534 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.345538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.345599 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.345710 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:57.346680 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.347295 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.348666 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.349087 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.349237 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.350440 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:57.350443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:57.350445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:57.350458 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:57.350476 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:57.350484 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:57.350563 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.350786 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.350852 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 00:00:57.351946 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.343182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38929 (local address 127.8.204.66:48415)
0807 00:00:57.343831 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.343871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.345325 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.345889 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.345915 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.346021 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:57.346538 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.346551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.347482 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.347486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.348532 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.348543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.349258 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.349261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.349403 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.350096 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:57.350099 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:57.350160 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:57.350263 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:57.350579 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:57.350588 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:57.350594 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.350786 (+ 192us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.351174 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260807 00:00:57.353170 2997 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38929
I20260807 00:00:57.353461 2997 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:57.354199 2997 heartbeater.cc:511] Master 127.8.204.126:38929 requested a full tablet report, sending...
I20260807 00:00:57.355517 2561 ts_manager.cc:194] Registered new tserver with Master: 7f4b41f34b614dde98ba25223d967bef (127.8.204.66:43315)
I20260807 00:00:57.356122 2561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.66:48415
W20260807 00:00:57.468983 3000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:57.469247 3000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:57.469286 3000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:57.473374 3000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:57.473431 3000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:57.473534 3000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:00:57.478399 3000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:38929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:57.479521 3000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:57.480922 3000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:57.490530 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:57.490530 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:57.490681 3000 server_base.cc:1034] running on GCE node
W20260807 00:00:57.490885 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:57.491307 3000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:57.491881 3000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:57.493055 3000 hybrid_clock.cc:648] HybridClock initialized: now 1786060857493037 us; error 34 us; skew 500 ppm
I20260807 00:00:57.495064 3000 webserver.cc:460] Webserver started at http://127.8.204.67:45719/ using document root <none> and password file <none>
I20260807 00:00:57.495662 3000 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:57.495764 3000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:57.496008 3000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:57.498044 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "9758a806d6354ecbbadb95be061e9992"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "11c21bd3e6d2aa8917b409b92bdc0e6d"
server_key_iv: "fd20705f721ce659d738fa4e4a4a7aa7"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.498612 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "9758a806d6354ecbbadb95be061e9992"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "11c21bd3e6d2aa8917b409b92bdc0e6d"
server_key_iv: "fd20705f721ce659d738fa4e4a4a7aa7"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.502202 3000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:00:57.504760 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.506060 3000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 00:00:57.506235 3000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "9758a806d6354ecbbadb95be061e9992"
format_stamp: "Formatted at 2026-08-07 00:00:57 on dist-test-slave-39l8"
server_key: "11c21bd3e6d2aa8917b409b92bdc0e6d"
server_key_iv: "fd20705f721ce659d738fa4e4a4a7aa7"
server_key_version: "encryptionkey@0"
I20260807 00:00:57.506352 3000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:57.516100 3000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:57.516772 3000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:57.516981 3000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:57.517596 3000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:57.518564 3000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:57.518639 3000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.518775 3000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:57.518813 3000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:57.528227 3000 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:33683
I20260807 00:00:57.528255 3179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:33683 every 8 connection(s)
I20260807 00:00:57.529162 3000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:00:57.536636 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 3000
I20260807 00:00:57.536791 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:00:57.537108 9009 external_mini_cluster.cc:1428] Setting key 3be831f9ccf880a33d9e239301f62447
I20260807 00:00:57.538496 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.531067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:34955 (local address 127.8.204.126:38929)
0807 00:00:57.531212 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.531216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.532261 (+ 1045us) server_negotiation.cc:407] Connection header received
0807 00:00:57.533674 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.533685 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.533748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.533851 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:57.534820 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.535345 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.536485 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.536715 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.536825 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.537972 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:57.537976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:57.537979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:57.537993 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:57.538016 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:57.538032 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:57.538115 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.538292 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.538355 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:00:57.539441 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.531437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:38929 (local address 127.8.204.67:34955)
0807 00:00:57.532093 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.532133 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.533460 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.534041 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.534047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.534164 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:57.534651 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.534664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.535475 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.535478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.536357 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.536366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.536880 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.536884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.537049 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.537735 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:00:57.537737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:57.537777 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:00:57.537842 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:57.538121 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:57.538132 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:57.538138 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.538278 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.538548 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260807 00:00:57.540468 3181 heartbeater.cc:347] Connected to a master server at 127.8.204.126:38929
I20260807 00:00:57.540709 3181 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:57.541229 3181 heartbeater.cc:511] Master 127.8.204.126:38929 requested a full tablet report, sending...
I20260807 00:00:57.542263 2561 ts_manager.cc:194] Registered new tserver with Master: 9758a806d6354ecbbadb95be061e9992 (127.8.204.67:33683)
I20260807 00:00:57.542927 2561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.67:34955
I20260807 00:00:57.551236 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:00:57.557085 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.553343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.8.204.126:38929)
0807 00:00:57.553480 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.553484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.553579 (+ 95us) server_negotiation.cc:407] Connection header received
0807 00:00:57.553675 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.553677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.553720 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.553800 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:57.554066 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.554607 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.555510 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.555735 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.555821 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.556643 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:57.556647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:57.556649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:00:57.556662 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:57.556681 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:00:57.556690 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:57.556795 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.556911 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.556942 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:00:57.560067 2561 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:00:57.563133 2561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:00:57.577494 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.571516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:43315 (local address 127.0.0.1:47154)
0807 00:00:57.571874 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.571924 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.572075 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.572992 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.572998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.573023 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:57.573178 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.573197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574657 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.574660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.575739 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.575751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.576930 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.576934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.577047 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.577102 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.577209 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.577333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260807 00:00:57.577682 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.571735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35481 (local address 127.0.0.1:32894)
0807 00:00:57.572075 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.572088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.572171 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.573390 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.573393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.573413 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:57.573491 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.573497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574864 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.574869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.576432 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.576442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.577279 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.577282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.577370 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.577407 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.577469 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.577539 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":309,"thread_start_us":92,"threads_started":1}
I20260807 00:00:57.577961 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.572494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.8.204.65:35481)
0807 00:00:57.572957 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.572969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.572990 (+ 21us) server_negotiation.cc:407] Connection header received
0807 00:00:57.573065 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.573070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.573257 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.573404 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:57.573624 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574748 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.576564 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.577125 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.577418 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.577606 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.577657 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.577778 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":131,"threads_started":1}
I20260807 00:00:57.578236 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.572222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:33683 (local address 127.0.0.1:53234)
0807 00:00:57.572533 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.572548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.572634 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.573736 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.573739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.573758 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:57.573842 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.573849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574949 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.574952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.576199 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.576211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.577230 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.577236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.577330 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.577940 (+ 610us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.578010 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.578094 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":92,"thread_start_us":98,"threads_started":1}
I20260807 00:00:57.578266 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.571675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.8.204.66:43315)
0807 00:00:57.572436 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.572448 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.572473 (+ 25us) server_negotiation.cc:407] Connection header received
0807 00:00:57.572554 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.572559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.572790 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.572990 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:57.573321 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574532 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.575894 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.576571 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.576676 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.576862 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.577994 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.578120 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":116,"threads_started":1}
I20260807 00:00:57.578725 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.572610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.8.204.67:33683)
0807 00:00:57.573294 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.573314 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.573337 (+ 23us) server_negotiation.cc:407] Connection header received
0807 00:00:57.573411 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.573415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.573601 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.573757 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:57.573977 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.574856 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.576340 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.577077 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.577204 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.577376 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.578449 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.578543 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":171,"threads_started":1}
I20260807 00:00:57.581041 2910 tablet_service.cc:1467] Processing CreateTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f (DEFAULT_TABLE table=test-table [id=c92f06a453d24d4fbe1a48013c347904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:57.581091 2740 tablet_service.cc:1467] Processing CreateTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f (DEFAULT_TABLE table=test-table [id=c92f06a453d24d4fbe1a48013c347904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:57.581087 3092 tablet_service.cc:1467] Processing CreateTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f (DEFAULT_TABLE table=test-table [id=c92f06a453d24d4fbe1a48013c347904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:00:57.581950 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95201eb67bc14c44bdee8d1d7556b66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:57.582091 3092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95201eb67bc14c44bdee8d1d7556b66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:57.582098 2740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95201eb67bc14c44bdee8d1d7556b66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:57.589226 3210 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Bootstrap starting.
I20260807 00:00:57.589226 3211 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Bootstrap starting.
I20260807 00:00:57.590426 3212 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Bootstrap starting.
I20260807 00:00:57.591746 3211 tablet_bootstrap.cc:654] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:57.591737 3210 tablet_bootstrap.cc:654] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:57.592288 3212 tablet_bootstrap.cc:654] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:57.592679 3210 log.cc:824] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:57.592679 3211 log.cc:824] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:57.593029 3212 log.cc:824] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:57.594836 3211 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: No bootstrap required, opened a new log
I20260807 00:00:57.594836 3210 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: No bootstrap required, opened a new log
I20260807 00:00:57.595037 3210 ts_tablet_manager.cc:1397] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 00:00:57.595037 3211 ts_tablet_manager.cc:1397] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 00:00:57.595561 3212 tablet_bootstrap.cc:492] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: No bootstrap required, opened a new log
I20260807 00:00:57.595775 3212 ts_tablet_manager.cc:1397] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 00:00:57.598850 3212 raft_consensus.cc:348] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.599090 3212 raft_consensus.cc:374] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:57.599154 3212 raft_consensus.cc:729] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda31eae3beb41d58c67ac8225bfe61e, State: Initialized, Role: FOLLOWER
I20260807 00:00:57.599377 3211 raft_consensus.cc:348] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.599377 3210 raft_consensus.cc:348] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.599618 3211 raft_consensus.cc:374] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:57.599618 3210 raft_consensus.cc:374] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:57.599677 3210 raft_consensus.cc:729] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4b41f34b614dde98ba25223d967bef, State: Initialized, Role: FOLLOWER
I20260807 00:00:57.599694 3211 raft_consensus.cc:729] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9758a806d6354ecbbadb95be061e9992, State: Initialized, Role: FOLLOWER
I20260807 00:00:57.599619 3212 consensus_queue.cc:260] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.600106 3211 consensus_queue.cc:260] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.600106 3210 consensus_queue.cc:260] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.600687 3212 ts_tablet_manager.cc:1428] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:00:57.600813 3181 heartbeater.cc:503] Master 127.8.204.126:38929 was elected leader, sending a full tablet report...
I20260807 00:00:57.601045 3210 ts_tablet_manager.cc:1428] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 00:00:57.601354 2997 heartbeater.cc:503] Master 127.8.204.126:38929 was elected leader, sending a full tablet report...
I20260807 00:00:57.601460 2824 heartbeater.cc:503] Master 127.8.204.126:38929 was elected leader, sending a full tablet report...
I20260807 00:00:57.602649 3211 ts_tablet_manager.cc:1428] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260807 00:00:57.648269 2825 tablet.cc:2369] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 00:00:57.780238 3182 tablet.cc:2369] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 00:00:57.842126 2998 tablet.cc:2369] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:00:57.917375 3217 raft_consensus.cc:482] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:00:57.917620 3217 raft_consensus.cc:504] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.919013 3217 leader_election.cc:288] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda31eae3beb41d58c67ac8225bfe61e (127.8.204.65:35481), 9758a806d6354ecbbadb95be061e9992 (127.8.204.67:33683)
I20260807 00:00:57.923499 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.919387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:50223 (local address 127.8.204.65:35481)
0807 00:00:57.919563 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.919569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.919743 (+ 174us) server_negotiation.cc:407] Connection header received
0807 00:00:57.919982 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.920002 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.920068 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.920153 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:57.920557 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.921170 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.922424 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.922929 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.923026 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.923068 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.923282 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.923343 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 00:00:57.923534 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.919275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35481 (local address 127.8.204.66:50223)
0807 00:00:57.919634 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.919648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.919797 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.920209 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.920211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.920236 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:57.920395 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.920422 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.921331 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.921334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.922286 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.922297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.923047 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.923052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.923141 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.923169 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.923257 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.923325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260807 00:00:57.923938 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.919800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:33683 (local address 127.8.204.66:48447)
0807 00:00:57.920212 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:57.920225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:57.920324 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:57.920576 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:57.920579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:57.920596 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:00:57.920721 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.920728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.921704 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.921706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.922567 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:57.922578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.923456 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.923460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:57.923568 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.923596 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 00:00:57.923724 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 00:00:57.923796 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":92,"threads_started":1}
I20260807 00:00:57.924010 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:57.919966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:48447 (local address 127.8.204.67:33683)
0807 00:00:57.920130 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:57.920135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:57.920318 (+ 183us) server_negotiation.cc:407] Connection header received
0807 00:00:57.920420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:57.920423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:57.920486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:57.920564 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:00:57.920893 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.921580 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.922742 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:57.923319 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:57.923436 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:57.923481 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:57.923757 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 00:00:57.923854 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 00:00:57.924382 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95201eb67bc14c44bdee8d1d7556b66f" candidate_uuid: "7f4b41f34b614dde98ba25223d967bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda31eae3beb41d58c67ac8225bfe61e" is_pre_election: true
I20260807 00:00:57.924669 2765 raft_consensus.cc:2454] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f4b41f34b614dde98ba25223d967bef in term 0.
I20260807 00:00:57.925185 2849 leader_election.cc:302] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f4b41f34b614dde98ba25223d967bef, dda31eae3beb41d58c67ac8225bfe61e; no voters:
I20260807 00:00:57.925450 3217 raft_consensus.cc:2781] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:00:57.925563 3217 raft_consensus.cc:482] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:00:57.925604 3217 raft_consensus.cc:3037] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:57.926066 3116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95201eb67bc14c44bdee8d1d7556b66f" candidate_uuid: "7f4b41f34b614dde98ba25223d967bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9758a806d6354ecbbadb95be061e9992" is_pre_election: true
I20260807 00:00:57.926364 3116 raft_consensus.cc:2454] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f4b41f34b614dde98ba25223d967bef in term 0.
I20260807 00:00:57.927132 3217 raft_consensus.cc:504] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.927485 3217 leader_election.cc:288] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [CANDIDATE]: Term 1 election: Requested vote from peers dda31eae3beb41d58c67ac8225bfe61e (127.8.204.65:35481), 9758a806d6354ecbbadb95be061e9992 (127.8.204.67:33683)
I20260807 00:00:57.927855 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95201eb67bc14c44bdee8d1d7556b66f" candidate_uuid: "7f4b41f34b614dde98ba25223d967bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda31eae3beb41d58c67ac8225bfe61e"
I20260807 00:00:57.927933 3116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95201eb67bc14c44bdee8d1d7556b66f" candidate_uuid: "7f4b41f34b614dde98ba25223d967bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9758a806d6354ecbbadb95be061e9992"
I20260807 00:00:57.928007 2765 raft_consensus.cc:3037] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:57.928074 3116 raft_consensus.cc:3037] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:57.929088 2765 raft_consensus.cc:2454] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f4b41f34b614dde98ba25223d967bef in term 1.
I20260807 00:00:57.929105 3116 raft_consensus.cc:2454] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f4b41f34b614dde98ba25223d967bef in term 1.
I20260807 00:00:57.929430 2849 leader_election.cc:302] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f4b41f34b614dde98ba25223d967bef, 9758a806d6354ecbbadb95be061e9992; no voters:
I20260807 00:00:57.929615 3217 raft_consensus.cc:2781] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:57.929831 3217 raft_consensus.cc:686] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 1 LEADER]: Becoming Leader. State: Replica: 7f4b41f34b614dde98ba25223d967bef, State: Running, Role: LEADER
I20260807 00:00:57.930209 3217 consensus_queue.cc:237] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } }
I20260807 00:00:57.933774 2560 catalog_manager.cc:5495] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f4b41f34b614dde98ba25223d967bef (127.8.204.66). New cstate: current_term: 1 leader_uuid: "7f4b41f34b614dde98ba25223d967bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4b41f34b614dde98ba25223d967bef" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 43315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 } health_report { overall_health: UNKNOWN } } }
I20260807 00:00:58.031652 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:58.027854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.8.204.66:43315)
0807 00:00:58.027984 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:58.027989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:58.028005 (+ 16us) server_negotiation.cc:407] Connection header received
0807 00:00:58.028130 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:58.028132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:58.028192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:58.028314 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:00:58.028577 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:58.029081 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:58.029951 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:58.030178 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:58.030292 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:58.030614 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:00:58.031184 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:00:58.031364 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:58.031458 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 00:00:58.031495 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 00:00:58.037855 2765 raft_consensus.cc:1261] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 1 FOLLOWER]: Refusing update from remote peer 7f4b41f34b614dde98ba25223d967bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:58.037855 3116 raft_consensus.cc:1261] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 1 FOLLOWER]: Refusing update from remote peer 7f4b41f34b614dde98ba25223d967bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:00:58.038597 3217 consensus_queue.cc:1035] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda31eae3beb41d58c67ac8225bfe61e" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:58.038836 3224 consensus_queue.cc:1035] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9758a806d6354ecbbadb95be061e9992" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:00:58.048161 3230 mvcc.cc:205] Tried to move back new op lower bound from 7315705274515185664 to 7315705274085203968. Current Snapshot: MvccSnapshot[applied={T|T < 7315705274515185664}]
I20260807 00:00:58.055600 2559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:58.055802 2559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:00:58.058485 2559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c2b9e1c31ee44da95449bbba1effbda: Sending DeleteTablet for 3 replicas of tablet 95201eb67bc14c44bdee8d1d7556b66f
I20260807 00:00:58.059321 2910 tablet_service.cc:1514] Processing DeleteTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:58 UTC) from {username='slave'} at 127.0.0.1:47154
I20260807 00:00:58.059471 2740 tablet_service.cc:1514] Processing DeleteTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:58 UTC) from {username='slave'} at 127.0.0.1:32894
I20260807 00:00:58.059561 3092 tablet_service.cc:1514] Processing DeleteTablet for tablet 95201eb67bc14c44bdee8d1d7556b66f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:00:58 UTC) from {username='slave'} at 127.0.0.1:53234
I20260807 00:00:58.059813 3241 tablet_replica.cc:330] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: stopping tablet replica
I20260807 00:00:58.060140 3241 raft_consensus.cc:2229] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 1 LEADER]: Raft consensus shutting down.
I20260807 00:00:58.060189 3244 tablet_replica.cc:330] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e: stopping tablet replica
I20260807 00:00:58.060491 3244 raft_consensus.cc:2229] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:58.060523 3241 raft_consensus.cc:2258] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:58.060843 3244 raft_consensus.cc:2258] T 95201eb67bc14c44bdee8d1d7556b66f P dda31eae3beb41d58c67ac8225bfe61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:58.060899 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 2653
I20260807 00:00:58.062089 3241 ts_tablet_manager.cc:1905] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:58.065034 3241 ts_tablet_manager.cc:1918] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:58.065124 3241 log.cc:1196] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/95201eb67bc14c44bdee8d1d7556b66f
I20260807 00:00:58.065455 3241 ts_tablet_manager.cc:1939] T 95201eb67bc14c44bdee8d1d7556b66f P 7f4b41f34b614dde98ba25223d967bef: Deleting consensus metadata
I20260807 00:00:58.066718 2539 catalog_manager.cc:4841] TS 7f4b41f34b614dde98ba25223d967bef (127.8.204.66:43315): tablet 95201eb67bc14c44bdee8d1d7556b66f (table test-table [id=c92f06a453d24d4fbe1a48013c347904]) successfully deleted
I20260807 00:00:58.067107 3245 tablet_replica.cc:330] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: stopping tablet replica
I20260807 00:00:58.067437 3245 raft_consensus.cc:2229] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:00:58.067759 3245 raft_consensus.cc:2258] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:00:58.070498 3245 ts_tablet_manager.cc:1905] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:00:58.073279 3245 ts_tablet_manager.cc:1918] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:00:58.073376 3245 log.cc:1196] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/95201eb67bc14c44bdee8d1d7556b66f
I20260807 00:00:58.073712 3245 ts_tablet_manager.cc:1939] T 95201eb67bc14c44bdee8d1d7556b66f P 9758a806d6354ecbbadb95be061e9992: Deleting consensus metadata
I20260807 00:00:58.079834 2540 catalog_manager.cc:4841] TS 9758a806d6354ecbbadb95be061e9992 (127.8.204.67:33683): tablet 95201eb67bc14c44bdee8d1d7556b66f (table test-table [id=c92f06a453d24d4fbe1a48013c347904]) successfully deleted
W20260807 00:00:58.091691 2540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:35481 (error 108)
I20260807 00:00:58.092604 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 2827
I20260807 00:00:58.096736 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:58.092575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:35481 (local address 127.0.0.1:32908)
0807 00:00:58.096504 (+ 3929us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:58.096596 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
W20260807 00:00:58.097105 2540 catalog_manager.cc:4584] TS dda31eae3beb41d58c67ac8225bfe61e (127.8.204.65:35481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95201eb67bc14c44bdee8d1d7556b66f: Network error: Client connection negotiation failed: client connection to 127.8.204.65:35481: connect: Connection refused (error 111)
I20260807 00:00:58.107784 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 3000
I20260807 00:00:58.124096 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 2519
2026-08-07T00:00:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1471 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:00:58 dist-test-slave-39l8 krb5kdc[3254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:00:58 dist-test-slave-39l8 krb5kdc[3254](info): set up 2 sockets
Aug 07 00:00:58 dist-test-slave-39l8 krb5kdc[3254](info): commencing operation
krb5kdc: starting...
W20260807 00:00:59.337332 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:00:59 dist-test-slave-39l8 krb5kdc[3254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:00:59.457348 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44383
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:59.569059 3276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:59.569301 3276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:59.569340 3276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:59.573342 3276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:00:59.573396 3276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:59.573415 3276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:59.573431 3276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:00:59.573446 3276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:00:59.578549 3276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:59.579520 3276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:59.580499 3276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 00:00:59.588853 3276 server_base.cc:1034] running on GCE node
W20260807 00:00:59.588819 3281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:59.588806 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:59.588809 3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:59.589636 3276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:59.590585 3276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:59.591784 3276 hybrid_clock.cc:648] HybridClock initialized: now 1786060859591764 us; error 30 us; skew 500 ppm
Aug 07 00:00:59 dist-test-slave-39l8 krb5kdc[3254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060859, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:59.599292 3276 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:00:59.600147 3276 webserver.cc:460] Webserver started at http://127.8.204.126:41791/ using document root <none> and password file <none>
I20260807 00:00:59.600663 3276 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:59.600719 3276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:59.600879 3276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:59.602926 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "7163e75282e473b5bc191d839d32c13b"
server_key_iv: "83295db2164a728827124aa7e5994a93"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.603442 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "7163e75282e473b5bc191d839d32c13b"
server_key_iv: "83295db2164a728827124aa7e5994a93"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.607127 3276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 00:00:59.609331 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:59.610484 3276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:00:59.610586 3276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "7163e75282e473b5bc191d839d32c13b"
server_key_iv: "83295db2164a728827124aa7e5994a93"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.610688 3276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:59.631093 3276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:59.634370 3276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:59.634524 3276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:59.642184 3276 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44383
I20260807 00:00:59.642210 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44383 every 8 connection(s)
I20260807 00:00:59.643453 3276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:00:59.646742 3378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:00:59.652308 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: Bootstrap starting.
I20260807 00:00:59.653453 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 3276
I20260807 00:00:59.653613 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:00:59.653918 9009 external_mini_cluster.cc:1428] Setting key 5b49cd78a8ce599f963337a9b718eb11
I20260807 00:00:59.655184 3378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: Neither blocks nor log segments found. Creating new log.
I20260807 00:00:59.656070 3378 log.cc:824] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: Log is configured to *not* fsync() on all Append() calls
I20260807 00:00:59.658272 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: No bootstrap required, opened a new log
I20260807 00:00:59.661823 3378 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } }
I20260807 00:00:59.662067 3378 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:00:59.662125 3378 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c0e45bc8824aeeb19f5ec0e31d28d5, State: Initialized, Role: FOLLOWER
I20260807 00:00:59.662616 3378 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } }
I20260807 00:00:59.662835 3378 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:00:59.662936 3378 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:00:59.663046 3378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:00:59.663889 3378 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } }
Aug 07 00:00:59 dist-test-slave-39l8 krb5kdc[3254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:00:59.664232 3378 leader_election.cc:302] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c0e45bc8824aeeb19f5ec0e31d28d5; no voters:
I20260807 00:00:59.664608 3378 leader_election.cc:288] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:00:59.664674 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:00:59.664883 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [term 1 LEADER]: Becoming Leader. State: Replica: b8c0e45bc8824aeeb19f5ec0e31d28d5, State: Running, Role: LEADER
I20260807 00:00:59.665185 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } }
I20260807 00:00:59.665877 3378 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:00:59.666368 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c0e45bc8824aeeb19f5ec0e31d28d5. Latest consensus state: current_term: 1 leader_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } } }
I20260807 00:00:59.666504 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:59.666815 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c0e45bc8824aeeb19f5ec0e31d28d5" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44383 } } }
I20260807 00:00:59.666927 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5 [sys.catalog]: This master's current role is: LEADER
I20260807 00:00:59.667683 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:00:59.671696 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:00:59.671845 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:59.655098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.8.204.126:44383)
0807 00:00:59.656823 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:59.656845 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:59.657219 (+ 374us) server_negotiation.cc:407] Connection header received
0807 00:00:59.657999 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:59.658015 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:59.658346 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:59.658727 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:59.659604 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.660538 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.661426 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.661888 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.662075 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:59.665131 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:59.665138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:59.665154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:59.665182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:59.668811 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:59.669352 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:59.669358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:59.669365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:59.669511 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:59.669812 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:59.669816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:59.669817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:59.670221 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:59.670450 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:59.670785 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 00:00:59.671103 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":1263,"threads_started":1}
I20260807 00:00:59.677443 3391 catalog_manager.cc:1347] Generated new cluster ID: 9e3fc79404754dcea49e8930c002de20
I20260807 00:00:59.677505 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:00:59.682514 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:00:59.683513 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:00:59.690444 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: Generated new TSK 0
I20260807 00:00:59.691133 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
Entry for principal kudu/127.8.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab.
I20260807 00:00:59.725337 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:00:59.853924 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:00:59.854197 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:00:59.854269 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:00:59.858174 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:00:59.858268 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:00:59.858352 3417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:00:59.863503 3417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.65.keytab
--principal=kudu/127.8.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:00:59.864480 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:00:59.865478 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:00:59.875229 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:59.875290 3417 server_base.cc:1034] running on GCE node
W20260807 00:00:59.875308 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:00:59.875265 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:00:59.875856 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:00:59.876402 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:00:59.877560 3417 hybrid_clock.cc:648] HybridClock initialized: now 1786060859877526 us; error 49 us; skew 500 ppm
Aug 07 00:00:59 dist-test-slave-39l8 krb5kdc[3254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060859, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:00:59.880373 3417 init.cc:377] Logged in from keytab as kudu/127.8.204.65@KRBTEST.COM (short username kudu)
I20260807 00:00:59.881203 3417 webserver.cc:460] Webserver started at http://127.8.204.65:46297/ using document root <none> and password file <none>
I20260807 00:00:59.881820 3417 fs_manager.cc:359] Metadata directory not provided
I20260807 00:00:59.881922 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:00:59.882158 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:00:59.884272 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "1edb859b952b490299ba4907317f5f1a"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "4146c06461f5a3458a3342496db566f9"
server_key_iv: "6255f5d40839981bfcb84255d868441d"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.884847 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "1edb859b952b490299ba4907317f5f1a"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "4146c06461f5a3458a3342496db566f9"
server_key_iv: "6255f5d40839981bfcb84255d868441d"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.888770 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 00:00:59.891196 3446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:59.892439 3417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:00:59.892573 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "1edb859b952b490299ba4907317f5f1a"
format_stamp: "Formatted at 2026-08-07 00:00:59 on dist-test-slave-39l8"
server_key: "4146c06461f5a3458a3342496db566f9"
server_key_iv: "6255f5d40839981bfcb84255d868441d"
server_key_version: "encryptionkey@0"
I20260807 00:00:59.892693 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:00:59.906334 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:00:59.909699 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:00:59.909912 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:00:59.910501 3417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:00:59.911381 3417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:00:59.911455 3417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:59.911523 3417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:00:59.911571 3417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:00:59.920697 3417 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:43287
I20260807 00:00:59.920725 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:43287 every 8 connection(s)
I20260807 00:00:59.921587 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:00:59.922576 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 3417
I20260807 00:00:59.922737 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:00:59.923059 9009 external_mini_cluster.cc:1428] Setting key 6b6cea4e4bdf896fa0196863479f4cd3
Aug 07 00:00:59 dist-test-slave-39l8 krb5kdc[3254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060859, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.65@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.66@KRBTEST.COM" created.
I20260807 00:00:59.937685 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:59.923566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41601 (local address 127.8.204.126:44383)
0807 00:00:59.923718 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:00:59.923721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:00:59.924763 (+ 1042us) server_negotiation.cc:407] Connection header received
0807 00:00:59.926147 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:00:59.926151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:00:59.926209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:00:59.926282 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:00:59.927995 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.928661 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.930509 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.930804 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.930891 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:59.933807 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:00:59.933810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:00:59.933812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:00:59.933835 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:00:59.935422 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:59.935952 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:59.935956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:59.935958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:59.936044 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:00:59.936474 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:00:59.936477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:00:59.936479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:00:59.936712 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:00:59.936829 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 00:00:59.937440 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 00:00:59.937549 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 00:00:59.938968 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:00:59.924020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44383 (local address 127.8.204.65:41601)
0807 00:00:59.924603 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0807 00:00:59.924660 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 00:00:59.925912 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:00:59.926472 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:00:59.926484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:00:59.927059 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:00:59.927796 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:59.927821 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.928803 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.928810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:59.930384 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:00:59.930400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:00:59.930976 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:00:59.930982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:00:59.931250 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:00:59.931904 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:00:59.931932 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:00:59.933622 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:00:59.935551 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:59.935557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:59.935571 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:59.935837 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:59.936162 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:00:59.936165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:00:59.936167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:00:59.936368 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:00:59.936829 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:00:59.936837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:00:59.937147 (+ 310us) client_negotiation.cc:769] Sending connection context
0807 00:00:59.937386 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 00:00:59.937754 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":152,"threads_started":1}
I20260807 00:00:59.940358 3620 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44383
I20260807 00:00:59.940629 3620 heartbeater.cc:464] Registering TS with master...
I20260807 00:00:59.941241 3620 heartbeater.cc:511] Master 127.8.204.126:44383 requested a full tablet report, sending...
I20260807 00:00:59.943153 3324 ts_manager.cc:194] Registered new tserver with Master: 1edb859b952b490299ba4907317f5f1a (127.8.204.65:43287)
I20260807 00:00:59.944800 3324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.65@KRBTEST.COM'} at 127.8.204.65:41601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
Entry for principal kudu/127.8.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab.
I20260807 00:00:59.953097 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:00.071441 3627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:00.071719 3627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:00.071765 3627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:00.075830 3627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:00.075893 3627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:00.075951 3627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:01:00.081372 3627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.66.keytab
--principal=kudu/127.8.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:00.082450 3627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:00.083546 3627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:00.093135 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:00.093133 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:00.093282 3627 server_base.cc:1034] running on GCE node
W20260807 00:01:00.093146 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:00.093859 3627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:00.094419 3627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:00.095584 3627 hybrid_clock.cc:648] HybridClock initialized: now 1786060860095563 us; error 29 us; skew 500 ppm
Aug 07 00:01:00 dist-test-slave-39l8 krb5kdc[3254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060860, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:00.098634 3627 init.cc:377] Logged in from keytab as kudu/127.8.204.66@KRBTEST.COM (short username kudu)
I20260807 00:01:00.099547 3627 webserver.cc:460] Webserver started at http://127.8.204.66:36897/ using document root <none> and password file <none>
I20260807 00:01:00.100135 3627 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:00.100231 3627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:00.100479 3627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:00.102535 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "7dbed20bb1914f2d9c9b847eff45cd5c"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5c91d2e6d1bc938021339cd21276edda"
server_key_iv: "4a02d70add0f101e4443e3f35970845b"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.103169 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "7dbed20bb1914f2d9c9b847eff45cd5c"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5c91d2e6d1bc938021339cd21276edda"
server_key_iv: "4a02d70add0f101e4443e3f35970845b"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.106950 3627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:01:00.109403 3650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.110599 3627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:00.110780 3627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "7dbed20bb1914f2d9c9b847eff45cd5c"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5c91d2e6d1bc938021339cd21276edda"
server_key_iv: "4a02d70add0f101e4443e3f35970845b"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.110898 3627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:00.131500 3627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:00.134994 3627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:00.135201 3627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:00.135823 3627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:01:00.136696 3627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:01:00.136770 3627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.136845 3627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:01:00.136894 3627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.146248 3627 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:42743
I20260807 00:01:00.146299 3810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:42743 every 8 connection(s)
I20260807 00:01:00.147276 3627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:01:00.150206 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 3627
I20260807 00:01:00.150359 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:01:00.150730 9009 external_mini_cluster.cc:1428] Setting key 76bbf8ccfb96b9aa0b19b6f8385cc7f0
Aug 07 00:01:00 dist-test-slave-39l8 krb5kdc[3254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060860, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.66@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.67@KRBTEST.COM; defaulting to no policy
I20260807 00:01:00.163271 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.149185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:54559 (local address 127.8.204.126:44383)
0807 00:01:00.149337 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.149341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.150368 (+ 1027us) server_negotiation.cc:407] Connection header received
0807 00:01:00.152070 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.152073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.152153 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.152250 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:00.154092 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.154712 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.155865 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.156268 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.156395 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.159136 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:00.159143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:00.159147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:00.159181 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:00.161067 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.161673 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.161677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.161678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.161776 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.162121 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.162128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.162131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.162365 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:00.162467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.163047 (+ 580us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.163149 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 00:01:00.164106 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.149563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44383 (local address 127.8.204.66:54559)
0807 00:01:00.150132 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.150198 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.151797 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.152662 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.152671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.153187 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:00.153891 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.153913 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.154865 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.154867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.155730 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.155739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.156420 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.156423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.156573 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.157208 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:00.157223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:00.158955 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:00.161229 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:00.161237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:00.161254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:00.161558 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:00.161880 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:00.161883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:00.161884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:00.162014 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:00.162470 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:00.162475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:00.162803 (+ 328us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.162971 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.163238 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.67@KRBTEST.COM" created.
I20260807 00:01:00.165369 3811 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44383
I20260807 00:01:00.165637 3811 heartbeater.cc:464] Registering TS with master...
I20260807 00:01:00.166198 3811 heartbeater.cc:511] Master 127.8.204.126:44383 requested a full tablet report, sending...
I20260807 00:01:00.167376 3324 ts_manager.cc:194] Registered new tserver with Master: 7dbed20bb1914f2d9c9b847eff45cd5c (127.8.204.66:42743)
I20260807 00:01:00.168088 3324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.66@KRBTEST.COM'} at 127.8.204.66:54559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
Entry for principal kudu/127.8.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab.
I20260807 00:01:00.179407 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:00.293244 3820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:00.293488 3820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:00.293566 3820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:00.297735 3820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:00.297799 3820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:00.297899 3820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:01:00.303224 3820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.67.keytab
--principal=kudu/127.8.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:00.304278 3820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:00.305308 3820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:00.315114 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:00.315125 3831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:00.315146 3828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:00.315425 3820 server_base.cc:1034] running on GCE node
I20260807 00:01:00.315846 3820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:00.316468 3820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:00.317654 3820 hybrid_clock.cc:648] HybridClock initialized: now 1786060860317642 us; error 29 us; skew 500 ppm
Aug 07 00:01:00 dist-test-slave-39l8 krb5kdc[3254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060860, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:00.320434 3820 init.cc:377] Logged in from keytab as kudu/127.8.204.67@KRBTEST.COM (short username kudu)
I20260807 00:01:00.321324 3820 webserver.cc:460] Webserver started at http://127.8.204.67:33637/ using document root <none> and password file <none>
I20260807 00:01:00.321913 3820 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:00.322006 3820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:00.322252 3820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:00.324393 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "9a532335c6444fd5996fd8f692f74e94"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "8b3ecbb3b8a1ff83df35809e0fca184c"
server_key_iv: "6619659b15626a7484b27843a1025556"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.324978 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "9a532335c6444fd5996fd8f692f74e94"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "8b3ecbb3b8a1ff83df35809e0fca184c"
server_key_iv: "6619659b15626a7484b27843a1025556"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.328801 3820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 00:01:00.331331 3841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.332564 3820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:00.332718 3820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "9a532335c6444fd5996fd8f692f74e94"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "8b3ecbb3b8a1ff83df35809e0fca184c"
server_key_iv: "6619659b15626a7484b27843a1025556"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.332831 3820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:00.356011 3820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:00.359061 3820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:00.359283 3820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:00.359920 3820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:01:00.360793 3820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:01:00.360869 3820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.360948 3820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:01:00.360992 3820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.370641 3820 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:45125
I20260807 00:01:00.370669 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:45125 every 8 connection(s)
I20260807 00:01:00.371594 3820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:01:00.376461 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 3820
I20260807 00:01:00.376596 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:01:00.376911 9009 external_mini_cluster.cc:1428] Setting key a114e199928bd5a9f51faab425e03266
Aug 07 00:01:00 dist-test-slave-39l8 krb5kdc[3254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060860, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.67@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:00.386023 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.373457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:48643 (local address 127.8.204.126:44383)
0807 00:01:00.373601 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.373605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.374475 (+ 870us) server_negotiation.cc:407] Connection header received
0807 00:01:00.375916 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.375920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.375981 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.376065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:00.377469 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.378234 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.379711 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.379919 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.380002 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.382381 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:00.382384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:00.382387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:00.382412 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:00.383965 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.384436 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.384439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.384440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.384551 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.384890 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.384893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.384894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.385071 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:00.385164 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.385690 (+ 526us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.385798 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:01:00.386883 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.373848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44383 (local address 127.8.204.67:48643)
0807 00:01:00.374325 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.374369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.375696 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.376235 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.376242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.376705 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:00.377298 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.377318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.378391 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.378395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.379592 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.379603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.380051 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.380054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.380197 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.380745 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:00.380761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:00.382225 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:00.384076 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:00.384082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:00.384095 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:00.384348 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:00.384663 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:00.384666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:00.384667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:00.384807 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:00.385168 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:00.385173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:00.385412 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.385584 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.385843 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":121,"threads_started":1}
I20260807 00:01:00.388250 4002 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44383
I20260807 00:01:00.388542 4002 heartbeater.cc:464] Registering TS with master...
I20260807 00:01:00.389084 4002 heartbeater.cc:511] Master 127.8.204.126:44383 requested a full tablet report, sending...
I20260807 00:01:00.390347 3322 ts_manager.cc:194] Registered new tserver with Master: 9a532335c6444fd5996fd8f692f74e94 (127.8.204.67:45125)
I20260807 00:01:00.391156 3322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.67@KRBTEST.COM'} at 127.8.204.67:48643
I20260807 00:01:00.391198 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:01:00.401042 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.393468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.8.204.126:44383)
0807 00:01:00.393590 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.393594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.393774 (+ 180us) server_negotiation.cc:407] Connection header received
0807 00:01:00.393866 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.393868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.393905 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.393967 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:00.394604 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.395159 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.395998 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.396207 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.396346 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.397679 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:00.397682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:00.397684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:00.397706 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:00.399229 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.399664 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.399671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.399674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.399759 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:00.400043 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:00.400047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:00.400049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:00.400272 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:00.400419 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.400657 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.400772 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 00:01:00.404095 3322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:01:00.407084 3322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:01:00.421888 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.416059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:43287 (local address 127.0.0.1:56560)
0807 00:01:00.417176 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.417189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.417358 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.418065 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.418068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.418083 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:00.418165 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.418170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419314 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.419318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.420534 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.420545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.421470 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.421473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.421557 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.421587 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.421683 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.421768 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":79,"thread_start_us":143,"threads_started":1}
I20260807 00:01:00.421978 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.416674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.8.204.65:43287)
0807 00:01:00.417133 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.417151 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.417280 (+ 129us) server_negotiation.cc:407] Connection header received
0807 00:01:00.417512 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.417518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.417725 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.417860 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:00.418297 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419140 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.420668 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.421342 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.421458 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.421631 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.421721 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.421819 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":123,"threads_started":1}
I20260807 00:01:00.422875 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.416273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:42743 (local address 127.0.0.1:44868)
0807 00:01:00.417248 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.417263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.417385 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.417876 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.417883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.417924 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:00.418195 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.418198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419672 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.419675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.421070 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.421082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.422409 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.422412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.422518 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.422548 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.422633 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.422697 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":96,"spinlock_wait_cycles":5760,"thread_start_us":89,"threads_started":1}
I20260807 00:01:00.423091 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.416829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.8.204.66:42743)
0807 00:01:00.417328 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.417340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.417373 (+ 33us) server_negotiation.cc:407] Connection header received
0807 00:01:00.417552 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.417557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.417714 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.417880 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:00.418330 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419505 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.421216 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.422153 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.422299 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.422443 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.422815 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.422923 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":154,"threads_started":1}
I20260807 00:01:00.425101 3525 tablet_service.cc:1467] Processing CreateTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a (DEFAULT_TABLE table=test-table [id=f4c261e21fd34c17a18332ab8d99fa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:00.425299 3714 tablet_service.cc:1467] Processing CreateTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a (DEFAULT_TABLE table=test-table [id=f4c261e21fd34c17a18332ab8d99fa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:00.426194 3525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:00.426251 3714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:00.428359 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.415837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45125 (local address 127.0.0.1:59144)
0807 00:01:00.416763 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.416817 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.416939 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.417884 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.417888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.417921 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:00.418094 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.418101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419672 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.419675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.421070 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.421082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.427905 (+ 6823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.427908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.428004 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.428031 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.428126 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.428205 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":147,"spinlock_wait_cycles":4096,"thread_start_us":86,"threads_started":1}
I20260807 00:01:00.428421 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.416020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.8.204.67:45125)
0807 00:01:00.417443 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.417458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.417479 (+ 21us) server_negotiation.cc:407] Connection header received
0807 00:01:00.417543 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.417548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.417725 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.418079 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:00.418254 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.419505 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.422512 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.427729 (+ 5217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.427904 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.428054 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.428169 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.428274 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":176,"threads_started":1}
I20260807 00:01:00.431488 3913 tablet_service.cc:1467] Processing CreateTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a (DEFAULT_TABLE table=test-table [id=f4c261e21fd34c17a18332ab8d99fa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:00.432665 3913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:00.433625 4030 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: Bootstrap starting.
I20260807 00:01:00.435029 4031 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Bootstrap starting.
I20260807 00:01:00.436141 4030 tablet_bootstrap.cc:654] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:00.437590 4030 log.cc:824] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:00.438046 4031 tablet_bootstrap.cc:654] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:00.438936 4031 log.cc:824] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:00.439533 4034 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Bootstrap starting.
I20260807 00:01:00.439982 4030 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: No bootstrap required, opened a new log
I20260807 00:01:00.440196 4030 ts_tablet_manager.cc:1397] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 00:01:00.440909 4031 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: No bootstrap required, opened a new log
I20260807 00:01:00.441105 4031 ts_tablet_manager.cc:1397] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260807 00:01:00.441547 4034 tablet_bootstrap.cc:654] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:00.442559 4034 log.cc:824] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:00.444222 4034 tablet_bootstrap.cc:492] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: No bootstrap required, opened a new log
I20260807 00:01:00.444429 4034 ts_tablet_manager.cc:1397] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 00:01:00.444254 4030 raft_consensus.cc:348] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.444535 4030 raft_consensus.cc:374] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:00.444599 4030 raft_consensus.cc:729] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edb859b952b490299ba4907317f5f1a, State: Initialized, Role: FOLLOWER
I20260807 00:01:00.445079 4030 consensus_queue.cc:260] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.445026 4031 raft_consensus.cc:348] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.445283 4031 raft_consensus.cc:374] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:00.445343 4031 raft_consensus.cc:729] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbed20bb1914f2d9c9b847eff45cd5c, State: Initialized, Role: FOLLOWER
I20260807 00:01:00.445802 4031 consensus_queue.cc:260] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.446141 4030 ts_tablet_manager.cc:1428] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 00:01:00.446516 3620 heartbeater.cc:503] Master 127.8.204.126:44383 was elected leader, sending a full tablet report...
I20260807 00:01:00.446532 3811 heartbeater.cc:503] Master 127.8.204.126:44383 was elected leader, sending a full tablet report...
I20260807 00:01:00.446750 4031 ts_tablet_manager.cc:1428] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260807 00:01:00.450109 4034 raft_consensus.cc:348] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.450405 4034 raft_consensus.cc:374] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:00.450470 4034 raft_consensus.cc:729] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a532335c6444fd5996fd8f692f74e94, State: Initialized, Role: FOLLOWER
I20260807 00:01:00.450950 4034 consensus_queue.cc:260] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.451678 4002 heartbeater.cc:503] Master 127.8.204.126:44383 was elected leader, sending a full tablet report...
I20260807 00:01:00.451949 4034 ts_tablet_manager.cc:1428] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 00:01:00.551187 4042 raft_consensus.cc:482] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:01:00.551420 4042 raft_consensus.cc:504] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.552770 4042 leader_election.cc:288] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a532335c6444fd5996fd8f692f74e94 (127.8.204.67:45125), 1edb859b952b490299ba4907317f5f1a (127.8.204.65:43287)
I20260807 00:01:00.556470 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.553053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:43287 (local address 127.8.204.66:50813)
0807 00:01:00.553216 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.553235 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.553345 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.553655 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.553658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.553676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:00.553797 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.553803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.554540 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.554545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.555359 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.555368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.556012 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.556016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.556132 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.556150 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.556249 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.556322 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 00:01:00.556469 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.553184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:50813 (local address 127.8.204.65:43287)
0807 00:01:00.553317 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.553321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.553337 (+ 16us) server_negotiation.cc:407] Connection header received
0807 00:01:00.553508 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.553511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.553554 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.553642 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:00.553911 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.554422 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.555494 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.555885 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.556013 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.556069 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.556269 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.556341 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 00:01:00.556936 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.553052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:45125 (local address 127.8.204.66:49753)
0807 00:01:00.553609 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.553621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:00.553756 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:00.554033 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:00.554036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:00.554053 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:00.554187 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.554192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.554971 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.554977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.555761 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:00.555770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.556460 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.556464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:00.556570 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.556608 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 00:01:00.556725 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 00:01:00.556790 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":144,"thread_start_us":176,"threads_started":1}
I20260807 00:01:00.556932 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.553149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:49753 (local address 127.8.204.67:45125)
0807 00:01:00.553281 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.553286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.553724 (+ 438us) server_negotiation.cc:407] Connection header received
0807 00:01:00.553880 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.553884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.553934 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.554021 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:00.554304 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.554859 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.555898 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.556327 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.556465 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.556516 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.556748 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.556822 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 00:01:00.557308 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614cd5e11c04e6fbdc1aaaaa0669c0a" candidate_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edb859b952b490299ba4907317f5f1a" is_pre_election: true
I20260807 00:01:00.557672 3934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614cd5e11c04e6fbdc1aaaaa0669c0a" candidate_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a532335c6444fd5996fd8f692f74e94" is_pre_election: true
I20260807 00:01:00.557693 3553 raft_consensus.cc:2454] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dbed20bb1914f2d9c9b847eff45cd5c in term 0.
I20260807 00:01:00.557946 3934 raft_consensus.cc:2454] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dbed20bb1914f2d9c9b847eff45cd5c in term 0.
I20260807 00:01:00.558306 3656 leader_election.cc:302] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1edb859b952b490299ba4907317f5f1a, 7dbed20bb1914f2d9c9b847eff45cd5c; no voters:
I20260807 00:01:00.558652 4042 raft_consensus.cc:2781] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:01:00.558777 4042 raft_consensus.cc:482] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:01:00.558856 4042 raft_consensus.cc:3037] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:00.559949 4042 raft_consensus.cc:504] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.560328 4042 leader_election.cc:288] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [CANDIDATE]: Term 1 election: Requested vote from peers 9a532335c6444fd5996fd8f692f74e94 (127.8.204.67:45125), 1edb859b952b490299ba4907317f5f1a (127.8.204.65:43287)
I20260807 00:01:00.560712 3934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614cd5e11c04e6fbdc1aaaaa0669c0a" candidate_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a532335c6444fd5996fd8f692f74e94"
I20260807 00:01:00.560750 3553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4614cd5e11c04e6fbdc1aaaaa0669c0a" candidate_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edb859b952b490299ba4907317f5f1a"
I20260807 00:01:00.560854 3934 raft_consensus.cc:3037] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:00.560880 3553 raft_consensus.cc:3037] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:00.561947 3553 raft_consensus.cc:2454] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dbed20bb1914f2d9c9b847eff45cd5c in term 1.
I20260807 00:01:00.561954 3934 raft_consensus.cc:2454] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dbed20bb1914f2d9c9b847eff45cd5c in term 1.
I20260807 00:01:00.562323 3656 leader_election.cc:302] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1edb859b952b490299ba4907317f5f1a, 7dbed20bb1914f2d9c9b847eff45cd5c; no voters:
I20260807 00:01:00.562589 4042 raft_consensus.cc:2781] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:00.562870 4042 raft_consensus.cc:686] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 1 LEADER]: Becoming Leader. State: Replica: 7dbed20bb1914f2d9c9b847eff45cd5c, State: Running, Role: LEADER
I20260807 00:01:00.563264 4042 consensus_queue.cc:237] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } }
I20260807 00:01:00.566844 3322 catalog_manager.cc:5495] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dbed20bb1914f2d9c9b847eff45cd5c (127.8.204.66). New cstate: current_term: 1 leader_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dbed20bb1914f2d9c9b847eff45cd5c" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 42743 } health_report { overall_health: HEALTHY } } }
I20260807 00:01:00.613358 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.609305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.8.204.66:42743)
0807 00:01:00.609463 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.609468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.609483 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:00.609647 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.609650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.609711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.609834 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:00.610128 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.610643 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.611625 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.611852 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.611995 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.612345 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:00.612852 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:00.613000 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.613071 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.613137 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 00:01:00.619638 3553 raft_consensus.cc:1261] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 1edb859b952b490299ba4907317f5f1a [term 1 FOLLOWER]: Refusing update from remote peer 7dbed20bb1914f2d9c9b847eff45cd5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:00.619634 3934 raft_consensus.cc:1261] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 1 FOLLOWER]: Refusing update from remote peer 7dbed20bb1914f2d9c9b847eff45cd5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:00.620409 4042 consensus_queue.cc:1035] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a532335c6444fd5996fd8f692f74e94" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:00.620646 4045 consensus_queue.cc:1035] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edb859b952b490299ba4907317f5f1a" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 00:01:00.622588 4004 tablet.cc:2369] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:01:00.628715 4050 mvcc.cc:205] Tried to move back new op lower bound from 7315705285090172928 to 7315705284869939200. Current Snapshot: MvccSnapshot[applied={T|T < 7315705285090172928}]
I20260807 00:01:00.629403 4052 mvcc.cc:205] Tried to move back new op lower bound from 7315705285090172928 to 7315705284869939200. Current Snapshot: MvccSnapshot[applied={T|T < 7315705285090172928}]
I20260807 00:01:00.639361 3322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:01:00.639596 3322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:01:00.642385 3322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c0e45bc8824aeeb19f5ec0e31d28d5: Sending DeleteTablet for 3 replicas of tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a
I20260807 00:01:00.643347 3913 tablet_service.cc:1514] Processing DeleteTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:59144
I20260807 00:01:00.643482 3525 tablet_service.cc:1514] Processing DeleteTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:56560
I20260807 00:01:00.644143 4066 tablet_replica.cc:330] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: stopping tablet replica
I20260807 00:01:00.644475 3714 tablet_service.cc:1514] Processing DeleteTablet for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:00 UTC) from {username='kudu', principal='kudu/127.8.204.126@KRBTEST.COM'} at 127.0.0.1:44868
I20260807 00:01:00.644665 4066 raft_consensus.cc:2229] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:01:00.645089 4066 raft_consensus.cc:2258] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:01:00.645493 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 3417
I20260807 00:01:00.651518 4066 ts_tablet_manager.cc:1905] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:01:00.654635 4066 ts_tablet_manager.cc:1918] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:01:00.654891 4066 log.cc:1196] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/4614cd5e11c04e6fbdc1aaaaa0669c0a
I20260807 00:01:00.655342 4066 ts_tablet_manager.cc:1939] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 9a532335c6444fd5996fd8f692f74e94: Deleting consensus metadata
I20260807 00:01:00.656896 3300 catalog_manager.cc:4841] TS 9a532335c6444fd5996fd8f692f74e94 (127.8.204.67:45125): tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a (table test-table [id=f4c261e21fd34c17a18332ab8d99fa7c]) successfully deleted
I20260807 00:01:00.657369 4068 tablet_replica.cc:330] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: stopping tablet replica
I20260807 00:01:00.657811 4068 raft_consensus.cc:2229] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 1 LEADER]: Raft consensus shutting down.
I20260807 00:01:00.658607 4068 raft_consensus.cc:2258] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:01:00.660483 4068 ts_tablet_manager.cc:1905] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:01:00.663563 4068 ts_tablet_manager.cc:1918] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:01:00.663740 4068 log.cc:1196] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/4614cd5e11c04e6fbdc1aaaaa0669c0a
I20260807 00:01:00.664142 4068 ts_tablet_manager.cc:1939] T 4614cd5e11c04e6fbdc1aaaaa0669c0a P 7dbed20bb1914f2d9c9b847eff45cd5c: Deleting consensus metadata
I20260807 00:01:00.665623 3298 catalog_manager.cc:4841] TS 7dbed20bb1914f2d9c9b847eff45cd5c (127.8.204.66:42743): tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a (table test-table [id=f4c261e21fd34c17a18332ab8d99fa7c]) successfully deleted
W20260807 00:01:00.666337 3301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:43287 (error 108)
I20260807 00:01:00.666522 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 3627
I20260807 00:01:00.672063 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.671675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:43287 (local address 127.0.0.1:56576)
0807 00:01:00.671884 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:00.671952 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260807 00:01:00.672254 3301 catalog_manager.cc:4584] TS 1edb859b952b490299ba4907317f5f1a (127.8.204.65:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4614cd5e11c04e6fbdc1aaaaa0669c0a: Network error: Client connection negotiation failed: client connection to 127.8.204.65:43287: connect: Connection refused (error 111)
I20260807 00:01:00.678786 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 3820
I20260807 00:01:00.694221 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 3276
2026-08-07T00:01:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2572 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T00:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:00Z Disabled control of system clock
I20260807 00:01:00.734411 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44999
--webserver_interface=127.8.204.126
--webserver_port=0
--builtin_ntp_servers=127.8.204.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:01:00.850145 4074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:00.850441 4074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:00.850478 4074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:00.854635 4074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:00.854733 4074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:00.854750 4074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:00.854768 4074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:00.854827 4074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:00.859678 4074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:44999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:00.860836 4074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:00.862179 4074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:00.870584 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:00.870543 4080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:00.870609 4082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:00.871156 4074 server_base.cc:1034] running on GCE node
I20260807 00:01:00.871788 4074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:00.872956 4074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:00.874162 4074 hybrid_clock.cc:648] HybridClock initialized: now 1786060860874148 us; error 37 us; skew 500 ppm
I20260807 00:01:00.876238 4074 webserver.cc:460] Webserver started at http://127.8.204.126:36769/ using document root <none> and password file <none>
I20260807 00:01:00.876861 4074 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:00.876966 4074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:00.877218 4074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:00.879676 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "64112601a21c44abbf12bf8f975e4a01"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5a55a265a01072b4a19be7841f9b5502"
server_key_iv: "4f6479f8a695c6a56d814bb5100d4d23"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.880343 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "64112601a21c44abbf12bf8f975e4a01"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5a55a265a01072b4a19be7841f9b5502"
server_key_iv: "4f6479f8a695c6a56d814bb5100d4d23"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.884218 4074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:01:00.886873 4091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:00.888068 4074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260807 00:01:00.888227 4074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "64112601a21c44abbf12bf8f975e4a01"
format_stamp: "Formatted at 2026-08-07 00:01:00 on dist-test-slave-39l8"
server_key: "5a55a265a01072b4a19be7841f9b5502"
server_key_iv: "4f6479f8a695c6a56d814bb5100d4d23"
server_key_version: "encryptionkey@0"
I20260807 00:01:00.888347 4074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:00.901733 4074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:00.902544 4074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:00.902822 4074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:00.910351 4074 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:44999
I20260807 00:01:00.910355 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:44999 every 8 connection(s)
I20260807 00:01:00.911520 4074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:00.914886 4168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:00.920600 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: Bootstrap starting.
I20260807 00:01:00.921010 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4074
I20260807 00:01:00.921140 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:00.921454 9009 external_mini_cluster.cc:1428] Setting key 707f884f8a3a589e8bb1cdae35b17f28
I20260807 00:01:00.923491 4168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:00.924263 4168 log.cc:824] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:00.926589 4168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: No bootstrap required, opened a new log
I20260807 00:01:00.931334 4168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } }
I20260807 00:01:00.931655 4168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:00.931732 4168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64112601a21c44abbf12bf8f975e4a01, State: Initialized, Role: FOLLOWER
I20260807 00:01:00.932262 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:00.922944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.8.204.126:44999)
0807 00:01:00.923680 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:00.923697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:00.923735 (+ 38us) server_negotiation.cc:407] Connection header received
0807 00:01:00.924455 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:00.924476 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:00.925003 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:00.925363 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:00.925743 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.926823 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.928050 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:00.928552 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:00.928676 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:00.929606 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:00.929611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:00.929626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:00.930031 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:00.930091 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:00.930111 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:00.930194 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:00.930387 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 00:01:00.930768 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260807 00:01:00.932291 4168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } }
I20260807 00:01:00.932427 4168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:00.932514 4168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:00.932612 4168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:00.933997 4168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } }
I20260807 00:01:00.934401 4168 leader_election.cc:302] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64112601a21c44abbf12bf8f975e4a01; no voters:
I20260807 00:01:00.934826 4168 leader_election.cc:288] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:00.935181 4174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:00.935431 4174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [term 1 LEADER]: Becoming Leader. State: Replica: 64112601a21c44abbf12bf8f975e4a01, State: Running, Role: LEADER
I20260807 00:01:00.935770 4174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } }
I20260807 00:01:00.936174 4168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:00.937664 4176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64112601a21c44abbf12bf8f975e4a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } } }
I20260807 00:01:00.937664 4178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64112601a21c44abbf12bf8f975e4a01. Latest consensus state: current_term: 1 leader_uuid: "64112601a21c44abbf12bf8f975e4a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64112601a21c44abbf12bf8f975e4a01" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 44999 } } }
I20260807 00:01:00.937821 4176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:00.937821 4178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:00.938314 4190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:00.941282 4190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:00.947120 4190 catalog_manager.cc:1347] Generated new cluster ID: dc68389c30ef4ec3b56d51adaa9527e9
I20260807 00:01:00.947206 4190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:00.973791 4190 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:00.975137 4190 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:00.981997 4190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: Generated new TSK 0
I20260807 00:01:00.982893 4190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:00.993062 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.65:0
--local_ip_for_outbound_sockets=127.8.204.65
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--builtin_ntp_servers=127.8.204.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 00:01:01.109514 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:01.109807 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:01.109875 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:01.113940 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:01.114027 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:01.114117 4199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.65
I20260807 00:01:01.118938 4199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.65
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.65
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:01.120062 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:01.121465 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:01.131132 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:01.131165 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:01.131208 4211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:01.131433 4199 server_base.cc:1034] running on GCE node
I20260807 00:01:01.131865 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:01.132571 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:01.133800 4199 hybrid_clock.cc:648] HybridClock initialized: now 1786060861133780 us; error 36 us; skew 500 ppm
I20260807 00:01:01.135694 4199 webserver.cc:460] Webserver started at http://127.8.204.65:35797/ using document root <none> and password file <none>
I20260807 00:01:01.136312 4199 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:01.136406 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:01.136694 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:01.139074 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data/instance:
uuid: "d7b4568a9ccc42a9b7df561528290b19"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "b386b0f8341b22cdb45c6c2f2107665b"
server_key_iv: "71c69afa96cc5e4e1cea266cb4b12eda"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.139668 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance:
uuid: "d7b4568a9ccc42a9b7df561528290b19"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "b386b0f8341b22cdb45c6c2f2107665b"
server_key_iv: "71c69afa96cc5e4e1cea266cb4b12eda"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.143309 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:01:01.145656 4220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.146936 4199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:01.147066 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal
uuid: "d7b4568a9ccc42a9b7df561528290b19"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "b386b0f8341b22cdb45c6c2f2107665b"
server_key_iv: "71c69afa96cc5e4e1cea266cb4b12eda"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.147179 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:01.156567 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:01.157226 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:01.157423 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:01.158003 4199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:01:01.159003 4199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:01:01.159076 4199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.159149 4199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:01:01.159192 4199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.168286 4199 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.65:36073
I20260807 00:01:01.168334 4385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.65:36073 every 8 connection(s)
I20260807 00:01:01.169214 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/data/info.pb
I20260807 00:01:01.170369 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4199
I20260807 00:01:01.170514 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-0/wal/instance
I20260807 00:01:01.170893 9009 external_mini_cluster.cc:1428] Setting key 99ac9ad21e3108e79e7646050b2d4c71
I20260807 00:01:01.172960 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.66:0
--local_ip_for_outbound_sockets=127.8.204.66
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--builtin_ntp_servers=127.8.204.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:01:01.179689 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.171386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:54733 (local address 127.8.204.126:44999)
0807 00:01:01.171534 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.171539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.172791 (+ 1252us) server_negotiation.cc:407] Connection header received
0807 00:01:01.174316 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.174322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.174389 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.174490 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:01.175799 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.176326 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.177536 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.177862 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.177992 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.179170 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:01.179173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:01.179176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:01.179190 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:01.179218 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:01.179227 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:01.179308 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.179468 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.179532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:01:01.180506 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.171833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44999 (local address 127.8.204.65:54733)
0807 00:01:01.172618 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.172669 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.174091 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.174762 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.174774 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.174892 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:01.175639 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.175660 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.176463 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.176466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.177399 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.177410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.178043 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.178048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.178223 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.178896 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:01:01.178898 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:01.178938 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:01:01.179002 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:01.179312 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:01.179324 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:01.179331 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.179467 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.179740 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260807 00:01:01.181896 4386 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44999
I20260807 00:01:01.182189 4386 heartbeater.cc:464] Registering TS with master...
I20260807 00:01:01.182907 4386 heartbeater.cc:511] Master 127.8.204.126:44999 requested a full tablet report, sending...
I20260807 00:01:01.184607 4112 ts_manager.cc:194] Registered new tserver with Master: d7b4568a9ccc42a9b7df561528290b19 (127.8.204.65:36073)
I20260807 00:01:01.185799 4112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.65:54733
W20260807 00:01:01.287539 4390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:01.287768 4390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:01.287830 4390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:01.291723 4390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:01.291802 4390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:01.291885 4390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.66
I20260807 00:01:01.296674 4390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.66
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.66
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:01.297646 4390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:01.299471 4390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:01.309473 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:01.309562 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:01.309487 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:01.309800 4390 server_base.cc:1034] running on GCE node
I20260807 00:01:01.310240 4390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:01.310916 4390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:01.312103 4390 hybrid_clock.cc:648] HybridClock initialized: now 1786060861312073 us; error 50 us; skew 500 ppm
I20260807 00:01:01.313868 4390 webserver.cc:460] Webserver started at http://127.8.204.66:33785/ using document root <none> and password file <none>
I20260807 00:01:01.314442 4390 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:01.314548 4390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:01.314843 4390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:01.316854 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data/instance:
uuid: "6339035435864966ad9a5bf1ec8c3c09"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "85ea9a3f722bb4d8f37781f02318b268"
server_key_iv: "ab64754920df9c978a170dd48484044b"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.317425 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance:
uuid: "6339035435864966ad9a5bf1ec8c3c09"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "85ea9a3f722bb4d8f37781f02318b268"
server_key_iv: "ab64754920df9c978a170dd48484044b"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.321159 4390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:01.323802 4410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.324986 4390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:01.325142 4390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal
uuid: "6339035435864966ad9a5bf1ec8c3c09"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "85ea9a3f722bb4d8f37781f02318b268"
server_key_iv: "ab64754920df9c978a170dd48484044b"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.325258 4390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:01.361287 4390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:01.362090 4390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:01.362298 4390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:01.362990 4390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:01:01.363984 4390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:01:01.364063 4390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.364137 4390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:01:01.364182 4390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.373394 4390 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.66:34201
I20260807 00:01:01.373437 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.66:34201 every 8 connection(s)
I20260807 00:01:01.374295 4390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/data/info.pb
I20260807 00:01:01.380894 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4390
I20260807 00:01:01.381055 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal/instance
I20260807 00:01:01.381357 9009 external_mini_cluster.cc:1428] Setting key afc0b01558019ef2d95dabda09329842
I20260807 00:01:01.383498 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.67:0
--local_ip_for_outbound_sockets=127.8.204.67
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--builtin_ntp_servers=127.8.204.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 00:01:01.384218 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.376198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.66:46505 (local address 127.8.204.126:44999)
0807 00:01:01.376393 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.376397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.377480 (+ 1083us) server_negotiation.cc:407] Connection header received
0807 00:01:01.379014 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.379018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.379071 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.379152 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:01.380058 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.380572 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.381927 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.382205 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.382308 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.383483 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:01.383487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:01.383490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:01.383503 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:01.383535 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:01.383553 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:01.383641 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.383875 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.383946 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 00:01:01.384984 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.376605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44999 (local address 127.8.204.66:46505)
0807 00:01:01.377330 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.377371 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.378677 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.379353 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.379359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.379439 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:01.379911 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.379923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.380724 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.380727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.381785 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.381795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.382368 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.382371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.382517 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.383227 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:01:01.383228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:01.383266 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:01:01.383327 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:01.383680 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:01.383694 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:01.383703 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.383858 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.384127 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":116,"threads_started":1}
I20260807 00:01:01.386152 4568 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44999
I20260807 00:01:01.386385 4568 heartbeater.cc:464] Registering TS with master...
I20260807 00:01:01.387023 4568 heartbeater.cc:511] Master 127.8.204.126:44999 requested a full tablet report, sending...
I20260807 00:01:01.388090 4112 ts_manager.cc:194] Registered new tserver with Master: 6339035435864966ad9a5bf1ec8c3c09 (127.8.204.66:34201)
I20260807 00:01:01.388666 4112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.66:46505
W20260807 00:01:01.497089 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:01.497328 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:01.497365 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:01.501377 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:01.501425 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:01.501509 4571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.67
I20260807 00:01:01.506132 4571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.67
--webserver_port=0
--tserver_master_addrs=127.8.204.126:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.67
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:01.507131 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:01.508435 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:01.517961 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:01.517936 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:01.518118 4571 server_base.cc:1034] running on GCE node
W20260807 00:01:01.517957 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:01.518558 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:01.519191 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:01.520373 4571 hybrid_clock.cc:648] HybridClock initialized: now 1786060861520355 us; error 29 us; skew 500 ppm
I20260807 00:01:01.522105 4571 webserver.cc:460] Webserver started at http://127.8.204.67:45615/ using document root <none> and password file <none>
I20260807 00:01:01.522670 4571 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:01.522784 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:01.523001 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:01.525117 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data/instance:
uuid: "a4f9b11bdba14f1495a5a65739af896f"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "715e83b4cd554748382c22a87527a44e"
server_key_iv: "117d2496f9ff3bba4297ae04d544f954"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.525679 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance:
uuid: "a4f9b11bdba14f1495a5a65739af896f"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "715e83b4cd554748382c22a87527a44e"
server_key_iv: "117d2496f9ff3bba4297ae04d544f954"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.529383 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 00:01:01.531803 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.533012 4571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:01:01.533150 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal
uuid: "a4f9b11bdba14f1495a5a65739af896f"
format_stamp: "Formatted at 2026-08-07 00:01:01 on dist-test-slave-39l8"
server_key: "715e83b4cd554748382c22a87527a44e"
server_key_iv: "117d2496f9ff3bba4297ae04d544f954"
server_key_version: "encryptionkey@0"
I20260807 00:01:01.533257 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:01.553138 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:01.553797 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:01.553987 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:01.554569 4571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 00:01:01.555567 4571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 00:01:01.555640 4571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.555716 4571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 00:01:01.555756 4571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:01.564854 4571 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.67:34735
I20260807 00:01:01.564904 4746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.67:34735 every 8 connection(s)
I20260807 00:01:01.565729 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/data/info.pb
I20260807 00:01:01.571120 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4571
I20260807 00:01:01.571254 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal/instance
I20260807 00:01:01.571592 9009 external_mini_cluster.cc:1428] Setting key 5b74a99ee77f6d62120608825f0d8e64
I20260807 00:01:01.577162 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.567641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.67:37177 (local address 127.8.204.126:44999)
0807 00:01:01.567808 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.567813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.568867 (+ 1054us) server_negotiation.cc:407] Connection header received
0807 00:01:01.570328 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.570334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.570395 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.570492 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:01.571718 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.572697 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.574551 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.574935 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.575069 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.576494 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:01.576499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:01.576502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:01.576517 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:01.576538 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:01.576547 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:01.576640 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.576897 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.576966 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 00:01:01.578514 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.568050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:44999 (local address 127.8.204.67:37177)
0807 00:01:01.568718 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.568757 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.570094 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.570721 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.570731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.570834 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:01.571523 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.571542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.572913 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.572931 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.574397 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.574410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.575123 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.575127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.575341 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.576156 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 00:01:01.576158 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:01.576228 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 00:01:01.576330 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:01.576675 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:01.576696 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:01.576704 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.576871 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.577230 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260807 00:01:01.580708 4747 heartbeater.cc:347] Connected to a master server at 127.8.204.126:44999
I20260807 00:01:01.580955 4747 heartbeater.cc:464] Registering TS with master...
I20260807 00:01:01.581599 4747 heartbeater.cc:511] Master 127.8.204.126:44999 requested a full tablet report, sending...
I20260807 00:01:01.582850 4112 ts_manager.cc:194] Registered new tserver with Master: a4f9b11bdba14f1495a5a65739af896f (127.8.204.67:34735)
I20260807 00:01:01.583503 4112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.67:37177
I20260807 00:01:01.587404 9009 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 00:01:01.592826 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.589324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.8.204.126:44999)
0807 00:01:01.589420 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.589437 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.589681 (+ 244us) server_negotiation.cc:407] Connection header received
0807 00:01:01.589767 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.589769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.589805 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.589868 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:01.590109 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.590574 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.591475 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.591680 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.591768 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.592505 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:01.592509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:01.592511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:01.592522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:01.592540 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:01.592547 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:01.592602 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.592669 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.592722 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 00:01:01.595498 4112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 00:01:01.598454 4112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 00:01:01.613237 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.608783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.8.204.66:34201)
0807 00:01:01.609139 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.609150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.609171 (+ 21us) server_negotiation.cc:407] Connection header received
0807 00:01:01.609255 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.609260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.609436 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.609580 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:01.609786 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.610893 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612004 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612582 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612719 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.612921 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.612982 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.613077 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260807 00:01:01.613262 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.607296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34735 (local address 127.0.0.1:52084)
0807 00:01:01.607654 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.607707 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.607842 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.608493 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.608499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.608519 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:01.608671 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.608682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.610310 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.610314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.611713 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.611725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612834 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.612949 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.612982 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.613029 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.613117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":57,"thread_start_us":122,"threads_started":1}
I20260807 00:01:01.613713 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.608054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:36073 (local address 127.0.0.1:44756)
0807 00:01:01.608310 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.608325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.608414 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.609355 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.609358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.609376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:01.609469 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.609479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.610746 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.610750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.611997 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.612006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612991 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.613401 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.613428 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.613503 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.613576 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":356,"thread_start_us":119,"threads_started":1}
I20260807 00:01:01.613879 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.608425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:34201 (local address 127.0.0.1:59040)
0807 00:01:01.608634 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.608647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.608727 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.609580 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.609583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.609605 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:01.609690 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.609697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.611009 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.611012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.611863 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.611875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612712 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.612820 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.612856 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.612959 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.613695 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":598,"thread_start_us":89,"threads_started":1}
I20260807 00:01:01.614187 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.607446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.8.204.67:34735)
0807 00:01:01.607977 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.607991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.608019 (+ 28us) server_negotiation.cc:407] Connection header received
0807 00:01:01.608098 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.608104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.608312 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.608850 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:01.609014 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.610180 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.611860 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612681 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.613665 (+ 984us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.613856 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.613910 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.614015 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":176,"threads_started":1}
I20260807 00:01:01.614190 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.608253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.8.204.65:36073)
0807 00:01:01.608918 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.608932 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.608958 (+ 26us) server_negotiation.cc:407] Connection header received
0807 00:01:01.609038 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.609046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.609214 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.609374 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:01.609646 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.610571 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.612124 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.612866 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.613114 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.613292 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.613899 (+ 607us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.614000 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":155,"threads_started":1}
I20260807 00:01:01.617072 4656 tablet_service.cc:1467] Processing CreateTablet for tablet 0281ccde86a943d6b73f6350f368a98b (DEFAULT_TABLE table=test-table [id=a2ac1142c22b4351b9c750b32046ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:01.617074 4283 tablet_service.cc:1467] Processing CreateTablet for tablet 0281ccde86a943d6b73f6350f368a98b (DEFAULT_TABLE table=test-table [id=a2ac1142c22b4351b9c750b32046ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:01.617067 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 0281ccde86a943d6b73f6350f368a98b (DEFAULT_TABLE table=test-table [id=a2ac1142c22b4351b9c750b32046ffec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 00:01:01.618216 4283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281ccde86a943d6b73f6350f368a98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:01.618216 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281ccde86a943d6b73f6350f368a98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:01.618238 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281ccde86a943d6b73f6350f368a98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:01.624606 4771 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Bootstrap starting.
I20260807 00:01:01.625955 4774 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Bootstrap starting.
I20260807 00:01:01.625955 4773 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Bootstrap starting.
I20260807 00:01:01.626971 4771 tablet_bootstrap.cc:654] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:01.627713 4771 log.cc:824] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:01.628172 4774 tablet_bootstrap.cc:654] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:01.628172 4773 tablet_bootstrap.cc:654] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:01.628999 4773 log.cc:824] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:01.628999 4774 log.cc:824] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:01.630398 4771 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: No bootstrap required, opened a new log
I20260807 00:01:01.630609 4771 ts_tablet_manager.cc:1397] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 00:01:01.630784 4774 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: No bootstrap required, opened a new log
I20260807 00:01:01.630980 4773 tablet_bootstrap.cc:492] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: No bootstrap required, opened a new log
I20260807 00:01:01.630987 4774 ts_tablet_manager.cc:1397] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:01:01.631181 4773 ts_tablet_manager.cc:1397] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:01:01.634007 4774 raft_consensus.cc:348] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.634264 4774 raft_consensus.cc:374] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:01.634318 4774 raft_consensus.cc:729] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4568a9ccc42a9b7df561528290b19, State: Initialized, Role: FOLLOWER
I20260807 00:01:01.634826 4774 consensus_queue.cc:260] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.635469 4386 heartbeater.cc:503] Master 127.8.204.126:44999 was elected leader, sending a full tablet report...
I20260807 00:01:01.635303 4773 raft_consensus.cc:348] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.635576 4773 raft_consensus.cc:374] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:01.635632 4773 raft_consensus.cc:729] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f9b11bdba14f1495a5a65739af896f, State: Initialized, Role: FOLLOWER
I20260807 00:01:01.635957 4774 ts_tablet_manager.cc:1428] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 00:01:01.635838 4771 raft_consensus.cc:348] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.636032 4773 consensus_queue.cc:260] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.636118 4771 raft_consensus.cc:374] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:01.636174 4771 raft_consensus.cc:729] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339035435864966ad9a5bf1ec8c3c09, State: Initialized, Role: FOLLOWER
I20260807 00:01:01.636590 4771 consensus_queue.cc:260] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.636787 4747 heartbeater.cc:503] Master 127.8.204.126:44999 was elected leader, sending a full tablet report...
I20260807 00:01:01.637014 4773 ts_tablet_manager.cc:1428] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 00:01:01.637506 4568 heartbeater.cc:503] Master 127.8.204.126:44999 was elected leader, sending a full tablet report...
I20260807 00:01:01.638383 4771 ts_tablet_manager.cc:1428] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260807 00:01:01.670390 4388 tablet.cc:2369] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 00:01:01.678264 4778 raft_consensus.cc:482] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 00:01:01.678503 4778 raft_consensus.cc:504] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.679785 4778 leader_election.cc:288] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4f9b11bdba14f1495a5a65739af896f (127.8.204.67:34735), 6339035435864966ad9a5bf1ec8c3c09 (127.8.204.66:34201)
I20260807 00:01:01.684015 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.680052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.67:34735 (local address 127.8.204.65:35905)
0807 00:01:01.680210 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.680227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.680347 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.680635 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.680638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.680666 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:01.680837 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.680845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.681739 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.681745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.682914 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.682924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.683577 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.683681 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.683715 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.683788 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.683868 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":66}
I20260807 00:01:01.684055 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.680231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:35905 (local address 127.8.204.67:34735)
0807 00:01:01.680375 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.680381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.680399 (+ 18us) server_negotiation.cc:407] Connection header received
0807 00:01:01.680459 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.680463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.680514 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.680599 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:01.680977 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.681617 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683051 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.683447 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683546 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.683626 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.683838 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.683919 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 00:01:01.684108 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.680459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.65:41835 (local address 127.8.204.66:34201)
0807 00:01:01.680632 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.680636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.680650 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:01:01.680803 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.680808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.680861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.680939 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 00:01:01.681221 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.681717 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683107 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.683486 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683626 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.683679 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.683903 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.683976 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 00:01:01.684275 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.680313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.66:34201 (local address 127.8.204.65:41835)
0807 00:01:01.680594 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.680607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:01.680682 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:01.680968 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:01.680971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:01.680989 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 00:01:01.681109 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.681115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.681849 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.681853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.682952 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:01.682963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.683654 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.683658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:01.683745 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.683763 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 00:01:01.683866 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 00:01:01.683931 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":14,"thread_start_us":94,"threads_started":1}
I20260807 00:01:01.685019 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281ccde86a943d6b73f6350f368a98b" candidate_uuid: "d7b4568a9ccc42a9b7df561528290b19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339035435864966ad9a5bf1ec8c3c09" is_pre_election: true
I20260807 00:01:01.685032 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281ccde86a943d6b73f6350f368a98b" candidate_uuid: "d7b4568a9ccc42a9b7df561528290b19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f9b11bdba14f1495a5a65739af896f" is_pre_election: true
I20260807 00:01:01.685340 4507 raft_consensus.cc:2454] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4568a9ccc42a9b7df561528290b19 in term 0.
I20260807 00:01:01.685354 4681 raft_consensus.cc:2454] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4568a9ccc42a9b7df561528290b19 in term 0.
I20260807 00:01:01.685880 4222 leader_election.cc:302] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6339035435864966ad9a5bf1ec8c3c09, d7b4568a9ccc42a9b7df561528290b19; no voters:
I20260807 00:01:01.686177 4778 raft_consensus.cc:2781] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 00:01:01.686275 4778 raft_consensus.cc:482] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 00:01:01.686322 4778 raft_consensus.cc:3037] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:01.687522 4778 raft_consensus.cc:504] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.687956 4778 leader_election.cc:288] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [CANDIDATE]: Term 1 election: Requested vote from peers a4f9b11bdba14f1495a5a65739af896f (127.8.204.67:34735), 6339035435864966ad9a5bf1ec8c3c09 (127.8.204.66:34201)
I20260807 00:01:01.688338 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281ccde86a943d6b73f6350f368a98b" candidate_uuid: "d7b4568a9ccc42a9b7df561528290b19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f9b11bdba14f1495a5a65739af896f"
I20260807 00:01:01.688372 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281ccde86a943d6b73f6350f368a98b" candidate_uuid: "d7b4568a9ccc42a9b7df561528290b19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339035435864966ad9a5bf1ec8c3c09"
I20260807 00:01:01.688452 4681 raft_consensus.cc:3037] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:01.688519 4507 raft_consensus.cc:3037] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:01.689610 4507 raft_consensus.cc:2454] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4568a9ccc42a9b7df561528290b19 in term 1.
I20260807 00:01:01.689658 4681 raft_consensus.cc:2454] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4568a9ccc42a9b7df561528290b19 in term 1.
I20260807 00:01:01.689970 4222 leader_election.cc:302] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6339035435864966ad9a5bf1ec8c3c09, d7b4568a9ccc42a9b7df561528290b19; no voters:
I20260807 00:01:01.690157 4778 raft_consensus.cc:2781] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:01.690402 4778 raft_consensus.cc:686] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [term 1 LEADER]: Becoming Leader. State: Replica: d7b4568a9ccc42a9b7df561528290b19, State: Running, Role: LEADER
I20260807 00:01:01.690798 4778 consensus_queue.cc:237] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } }
I20260807 00:01:01.694195 4110 catalog_manager.cc:5495] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b4568a9ccc42a9b7df561528290b19 (127.8.204.65). New cstate: current_term: 1 leader_uuid: "d7b4568a9ccc42a9b7df561528290b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4568a9ccc42a9b7df561528290b19" member_type: VOTER last_known_addr { host: "127.8.204.65" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 } health_report { overall_health: UNKNOWN } } }
I20260807 00:01:01.710201 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.706544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.8.204.65:36073)
0807 00:01:01.706699 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:01.706720 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:01.706735 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:01.706836 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:01.706839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:01.706895 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:01.707012 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:01.707254 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.707770 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.708646 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:01.708869 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:01.708978 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:01.709302 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:01.709737 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:01.709880 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:01.709923 (+ 43us) server_negotiation.cc:299] Negotiation successful
0807 00:01:01.709973 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 00:01:01.716162 4681 raft_consensus.cc:1261] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 1 FOLLOWER]: Refusing update from remote peer d7b4568a9ccc42a9b7df561528290b19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:01.716208 4507 raft_consensus.cc:1261] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 1 FOLLOWER]: Refusing update from remote peer d7b4568a9ccc42a9b7df561528290b19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:01.716881 4778 consensus_queue.cc:1035] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6339035435864966ad9a5bf1ec8c3c09" member_type: VOTER last_known_addr { host: "127.8.204.66" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:01.717067 4784 consensus_queue.cc:1035] T 0281ccde86a943d6b73f6350f368a98b P d7b4568a9ccc42a9b7df561528290b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f9b11bdba14f1495a5a65739af896f" member_type: VOTER last_known_addr { host: "127.8.204.67" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:01.725682 4790 mvcc.cc:205] Tried to move back new op lower bound from 7315705289582235648 to 7315705289488338944. Current Snapshot: MvccSnapshot[applied={T|T < 7315705289582235648}]
I20260807 00:01:01.727195 4789 mvcc.cc:205] Tried to move back new op lower bound from 7315705289582235648 to 7315705289488338944. Current Snapshot: MvccSnapshot[applied={T|T < 7315705289582235648}]
I20260807 00:01:01.733245 4112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:01:01.733384 4112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 00:01:01.735664 4112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64112601a21c44abbf12bf8f975e4a01: Sending DeleteTablet for 3 replicas of tablet 0281ccde86a943d6b73f6350f368a98b
I20260807 00:01:01.736462 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281ccde86a943d6b73f6350f368a98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:01 UTC) from {username='slave'} at 127.0.0.1:52084
I20260807 00:01:01.736555 4479 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281ccde86a943d6b73f6350f368a98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:01 UTC) from {username='slave'} at 127.0.0.1:59040
I20260807 00:01:01.737008 4800 tablet_replica.cc:330] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: stopping tablet replica
I20260807 00:01:01.737006 4799 tablet_replica.cc:330] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: stopping tablet replica
I20260807 00:01:01.737031 4283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281ccde86a943d6b73f6350f368a98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 00:01:01 UTC) from {username='slave'} at 127.0.0.1:44756
I20260807 00:01:01.737394 4799 raft_consensus.cc:2229] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:01:01.737413 4800 raft_consensus.cc:2229] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 00:01:01.737728 4799 raft_consensus.cc:2258] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:01:01.737870 4800 raft_consensus.cc:2258] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 00:01:01.738058 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4199
I20260807 00:01:01.741758 4799 ts_tablet_manager.cc:1905] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:01:01.744730 4800 ts_tablet_manager.cc:1905] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 00:01:01.745972 4799 ts_tablet_manager.cc:1918] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:01:01.746097 4799 log.cc:1196] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-2/wal/wals/0281ccde86a943d6b73f6350f368a98b
I20260807 00:01:01.746452 4799 ts_tablet_manager.cc:1939] T 0281ccde86a943d6b73f6350f368a98b P a4f9b11bdba14f1495a5a65739af896f: Deleting consensus metadata
I20260807 00:01:01.747717 4800 ts_tablet_manager.cc:1918] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 00:01:01.747884 4800 log.cc:1196] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Deleting WAL directory at /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060609240246-9009-0/minicluster-data/ts-1/wal/wals/0281ccde86a943d6b73f6350f368a98b
I20260807 00:01:01.747860 4092 catalog_manager.cc:4841] TS a4f9b11bdba14f1495a5a65739af896f (127.8.204.67:34735): tablet 0281ccde86a943d6b73f6350f368a98b (table test-table [id=a2ac1142c22b4351b9c750b32046ffec]) successfully deleted
I20260807 00:01:01.748263 4800 ts_tablet_manager.cc:1939] T 0281ccde86a943d6b73f6350f368a98b P 6339035435864966ad9a5bf1ec8c3c09: Deleting consensus metadata
I20260807 00:01:01.749303 4092 catalog_manager.cc:4841] TS 6339035435864966ad9a5bf1ec8c3c09 (127.8.204.66:34201): tablet 0281ccde86a943d6b73f6350f368a98b (table test-table [id=a2ac1142c22b4351b9c750b32046ffec]) successfully deleted
W20260807 00:01:01.752600 4092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.65:36073 (error 108)
I20260807 00:01:01.753419 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4390
I20260807 00:01:01.753672 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:01.753355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.65:36073 (local address 127.0.0.1:44778)
0807 00:01:01.753499 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:01.753578 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.65:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 00:01:01.757164 4092 catalog_manager.cc:4584] TS d7b4568a9ccc42a9b7df561528290b19 (127.8.204.65:36073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0281ccde86a943d6b73f6350f368a98b: Network error: Client connection negotiation failed: client connection to 127.8.204.65:36073: connect: Connection refused (error 111)
I20260807 00:01:01.764961 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4571
I20260807 00:01:01.776485 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4074
2026-08-07T00:01:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1077 ms)
[----------] 6 tests from EncryptionPolicyTest (11730 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:01 dist-test-slave-39l8 krb5kdc[4805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:01 dist-test-slave-39l8 krb5kdc[4805](info): set up 2 sockets
Aug 07 00:01:01 dist-test-slave-39l8 krb5kdc[4805](info): commencing operation
krb5kdc: starting...
W20260807 00:01:02.963207 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:03.085611 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:03.199450 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:03.199752 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:03.199872 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:03.203774 4826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:03.203858 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:03.203897 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:03.203933 4826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:03.203963 4826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:03.208981 4826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:03.209988 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:03.210996 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:03.219270 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:03.219298 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:03.219388 4826 server_base.cc:1034] running on GCE node
W20260807 00:01:03.219303 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:03.220075 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:03.221129 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:03.222321 4826 hybrid_clock.cc:648] HybridClock initialized: now 1786060863222293 us; error 42 us; skew 500 ppm
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:03.225057 4826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:03.225943 4826 webserver.cc:460] Webserver started at http://127.0.0.1:39905/ using document root <none> and password file <none>
I20260807 00:01:03.226497 4826 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:03.226588 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:03.226887 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:03.228879 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "13692ae6dffd487b9808d8d3c9778607"
format_stamp: "Formatted at 2026-08-07 00:01:03 on dist-test-slave-39l8"
server_key: "ef4612e7b09028d6f2f4e9964826f8d4"
server_key_iv: "f1c8e4a1500c012a8250476e234aeb48"
server_key_version: "encryptionkey@0"
I20260807 00:01:03.229453 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "13692ae6dffd487b9808d8d3c9778607"
format_stamp: "Formatted at 2026-08-07 00:01:03 on dist-test-slave-39l8"
server_key: "ef4612e7b09028d6f2f4e9964826f8d4"
server_key_iv: "f1c8e4a1500c012a8250476e234aeb48"
server_key_version: "encryptionkey@0"
I20260807 00:01:03.233394 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 00:01:03.236004 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:03.237222 4826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:01:03.237373 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "13692ae6dffd487b9808d8d3c9778607"
format_stamp: "Formatted at 2026-08-07 00:01:03 on dist-test-slave-39l8"
server_key: "ef4612e7b09028d6f2f4e9964826f8d4"
server_key_iv: "f1c8e4a1500c012a8250476e234aeb48"
server_key_version: "encryptionkey@0"
I20260807 00:01:03.237495 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:03.250887 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:03.253861 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:03.254067 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:03.261668 4826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40753
I20260807 00:01:03.261690 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40753 every 8 connection(s)
I20260807 00:01:03.262840 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:03.265720 4928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:03.271543 4928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: Bootstrap starting.
I20260807 00:01:03.272315 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4826
I20260807 00:01:03.272449 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:03.272744 9009 external_mini_cluster.cc:1428] Setting key c56c38cd9aba02fcd8dec3bc620cd2fe
I20260807 00:01:03.274533 4928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:03.275321 4928 log.cc:824] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:03.277315 4928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: No bootstrap required, opened a new log
I20260807 00:01:03.281052 4928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260807 00:01:03.281363 4928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:03.281430 4928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13692ae6dffd487b9808d8d3c9778607, State: Initialized, Role: FOLLOWER
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 00:01:03.281898 4928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260807 00:01:03.282025 4928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:03.282125 4928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:03.282259 4928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:03.283483 4928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260807 00:01:03.283840 4928 leader_election.cc:302] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13692ae6dffd487b9808d8d3c9778607; no voters:
I20260807 00:01:03.284179 4928 leader_election.cc:288] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:03.284300 4934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:03.284526 4934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [term 1 LEADER]: Becoming Leader. State: Replica: 13692ae6dffd487b9808d8d3c9778607, State: Running, Role: LEADER
I20260807 00:01:03.284865 4934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260807 00:01:03.285624 4928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:03.286942 4935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13692ae6dffd487b9808d8d3c9778607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260807 00:01:03.287072 4935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:03.288296 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:03.273972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35005 (local address 127.0.0.1:40753)
0807 00:01:03.274427 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:03.274442 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:03.274477 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:03.275202 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:03.275221 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:03.275608 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:03.275947 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:03.276819 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:03.277861 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:03.278745 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:03.279217 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:03.279402 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:03.282347 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:03.282356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:03.282374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:03.282413 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:03.284306 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:03.284979 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:03.284985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:03.284992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:03.285129 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:03.285572 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:03.285575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:03.285577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:03.286101 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:03.286325 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:03.287025 (+ 700us) server_negotiation.cc:299] Negotiation successful
0807 00:01:03.287287 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260807 00:01:03.288023 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13692ae6dffd487b9808d8d3c9778607. Latest consensus state: current_term: 1 leader_uuid: "13692ae6dffd487b9808d8d3c9778607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13692ae6dffd487b9808d8d3c9778607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260807 00:01:03.288383 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607 [sys.catalog]: This master's current role is: LEADER
W20260807 00:01:03.291239 4954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:03.291415 4954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:03.291544 4946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:03.291615 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:03.292264 4946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:03.298965 4946 catalog_manager.cc:1347] Generated new cluster ID: 5bf690c46d65478cb5e07fe89f184760
I20260807 00:01:03.299041 4946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:03.302518 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:03.293965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40609 (local address 127.0.0.1:40753)
0807 00:01:03.294078 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:03.294082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:03.294181 (+ 99us) server_negotiation.cc:407] Connection header received
0807 00:01:03.294371 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:03.294374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:03.294422 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:03.294505 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:03.295250 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:03.295784 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:03.296733 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:03.297075 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:03.297198 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:03.298846 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:03.298850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:03.298853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:03.298879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:03.300601 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:03.301179 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:03.301182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:03.301184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:03.301274 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:03.301642 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:03.301645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:03.301646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:03.301851 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:03.301992 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:03.302272 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 00:01:03.302367 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 00:01:03.317323 4946 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:03.318586 4946 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:03.324002 4946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13692ae6dffd487b9808d8d3c9778607: Generated new TSK 0
I20260807 00:01:03.324793 4946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:03.328758 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4826
W20260807 00:01:03.341789 4803 connection.cc:535] client connection to 127.0.0.1:40753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40753): Connection reset by peer (error 104)
2026-08-07T00:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4968](info): set up 2 sockets
Aug 07 00:01:03 dist-test-slave-39l8 krb5kdc[4968](info): commencing operation
krb5kdc: starting...
W20260807 00:01:04.529286 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:04 dist-test-slave-39l8 krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:04.651993 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:04.764820 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:04.765100 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:04.765182 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:04.769131 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:04.769223 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:04.769263 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:04.769321 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:04.769361 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:04.774446 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:04.775477 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:04.776451 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:04.784762 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:04.784818 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:04.784835 4985 server_base.cc:1034] running on GCE node
W20260807 00:01:04.785027 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:04.785512 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:04.786511 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:04.787724 4985 hybrid_clock.cc:648] HybridClock initialized: now 1786060864787698 us; error 38 us; skew 500 ppm
Aug 07 00:01:04 dist-test-slave-39l8 krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:04.799662 4985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:04.800552 4985 webserver.cc:460] Webserver started at http://127.0.0.1:42603/ using document root <none> and password file <none>
I20260807 00:01:04.801137 4985 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:04.801249 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:04.801518 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:04.803736 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "2edff3856e33483980bb0b2f122b21ec"
format_stamp: "Formatted at 2026-08-07 00:01:04 on dist-test-slave-39l8"
server_key: "c9daaf68ed32c8dd5197705feab904f2"
server_key_iv: "3a753eba907ef4ebff9da7b440eebc82"
server_key_version: "encryptionkey@0"
I20260807 00:01:04.804353 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "2edff3856e33483980bb0b2f122b21ec"
format_stamp: "Formatted at 2026-08-07 00:01:04 on dist-test-slave-39l8"
server_key: "c9daaf68ed32c8dd5197705feab904f2"
server_key_iv: "3a753eba907ef4ebff9da7b440eebc82"
server_key_version: "encryptionkey@0"
I20260807 00:01:04.808246 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 00:01:04.810774 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:04.811957 4985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:04.812121 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "2edff3856e33483980bb0b2f122b21ec"
format_stamp: "Formatted at 2026-08-07 00:01:04 on dist-test-slave-39l8"
server_key: "c9daaf68ed32c8dd5197705feab904f2"
server_key_iv: "3a753eba907ef4ebff9da7b440eebc82"
server_key_version: "encryptionkey@0"
I20260807 00:01:04.812239 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:04.829702 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:04.832948 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:04.833173 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:04.840291 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33001
I20260807 00:01:04.840309 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33001 every 8 connection(s)
I20260807 00:01:04.841318 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:04.844048 5081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:04.848899 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 4985
I20260807 00:01:04.849056 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:04.849345 9009 external_mini_cluster.cc:1428] Setting key e3f08542c718e2f77bbd5a75c0932ed8
I20260807 00:01:04.849570 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: Bootstrap starting.
I20260807 00:01:04.852171 5081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:04.852898 5081 log.cc:824] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:04.854681 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: No bootstrap required, opened a new log
I20260807 00:01:04.857985 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:04.850604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34093 (local address 127.0.0.1:33001)
0807 00:01:04.851066 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:04.851087 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:04.851122 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:04.851782 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:04.851799 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:04.852134 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:04.852492 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:04.852889 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:04.853892 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:04.854829 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:04.855280 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:04.855499 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:04.856356 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:04.856369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:04.856381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:04.856793 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:04.856843 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:04.856865 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:04.857007 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:04.857168 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 00:01:04.857304 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":176,"threads_started":1}
I20260807 00:01:04.858171 5081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } }
I20260807 00:01:04.858400 5081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:04.858476 5081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edff3856e33483980bb0b2f122b21ec, State: Initialized, Role: FOLLOWER
I20260807 00:01:04.859002 5081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } }
I20260807 00:01:04.859162 5081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:04.859233 5081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:04.859324 5081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:04.860347 5081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } }
I20260807 00:01:04.860724 5081 leader_election.cc:302] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edff3856e33483980bb0b2f122b21ec; no voters:
I20260807 00:01:04.861028 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:04.861090 5081 leader_election.cc:288] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:04.861208 5091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:04.861451 5091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [term 1 LEADER]: Becoming Leader. State: Replica: 2edff3856e33483980bb0b2f122b21ec, State: Running, Role: LEADER
I20260807 00:01:04.861801 5091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } }
I20260807 00:01:04.862366 5081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:04.863030 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edff3856e33483980bb0b2f122b21ec. Latest consensus state: current_term: 1 leader_uuid: "2edff3856e33483980bb0b2f122b21ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } } }
I20260807 00:01:04.863152 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:04.863471 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2edff3856e33483980bb0b2f122b21ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edff3856e33483980bb0b2f122b21ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33001 } } }
I20260807 00:01:04.863536 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec [sys.catalog]: This master's current role is: LEADER
W20260807 00:01:04.867286 5120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:04.867362 5120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:04.867434 5108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:04.868398 5108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:04.869221 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:04.864936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43861 (local address 127.0.0.1:33001)
0807 00:01:04.865034 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:04.865038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:04.865362 (+ 324us) server_negotiation.cc:407] Connection header received
0807 00:01:04.865560 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:04.865563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:04.865604 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:04.865701 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:04.865968 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:04.866471 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:04.867366 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:04.867674 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:04.867788 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:04.868695 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:04.868699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:04.868701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:04.868714 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:04.868741 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:04.868754 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:04.868848 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:04.869056 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 00:01:04.869098 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 00:01:04.873628 5108 catalog_manager.cc:1347] Generated new cluster ID: 6ad6186be3b34502b6dc7bfb6848a3d2
I20260807 00:01:04.873700 5108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:04.889484 5108 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:04.890479 5108 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:04.895324 5108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2edff3856e33483980bb0b2f122b21ec: Generated new TSK 0
I20260807 00:01:04.896011 5108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:04.906559 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 4985
W20260807 00:01:04.917380 4965 connection.cc:535] client connection to 127.0.0.1:33001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33001): Connection reset by peer (error 104)
2026-08-07T00:01:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1575 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:04 dist-test-slave-39l8 krb5kdc[5129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:04 dist-test-slave-39l8 krb5kdc[5129](info): set up 2 sockets
Aug 07 00:01:04 dist-test-slave-39l8 krb5kdc[5129](info): commencing operation
krb5kdc: starting...
W20260807 00:01:06.112038 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:06 dist-test-slave-39l8 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:06.235591 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:06.346508 5145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:06.346845 5145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:06.346953 5145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:06.350947 5145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:06.351024 5145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:06.351061 5145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:06.351120 5145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:06.351157 5145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:06.356525 5145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:06.357625 5145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:06.358654 5145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:06.367120 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:06.367242 5145 server_base.cc:1034] running on GCE node
W20260807 00:01:06.367122 5158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:06.367080 5152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:06.367846 5145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:06.368937 5145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:06.370141 5145 hybrid_clock.cc:648] HybridClock initialized: now 1786060866370121 us; error 38 us; skew 500 ppm
Aug 07 00:01:06 dist-test-slave-39l8 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:06.372936 5145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:06.373824 5145 webserver.cc:460] Webserver started at http://127.0.0.1:44819/ using document root <none> and password file <none>
I20260807 00:01:06.374390 5145 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:06.374480 5145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:06.374763 5145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:06.376776 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "f7781d35666247ad85fea2b427743abf"
format_stamp: "Formatted at 2026-08-07 00:01:06 on dist-test-slave-39l8"
server_key: "f750a6bb37e399f9d22a55210105efc8"
server_key_iv: "951e9a5a1248972de9ac20dbbf234eb4"
server_key_version: "encryptionkey@0"
I20260807 00:01:06.377347 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "f7781d35666247ad85fea2b427743abf"
format_stamp: "Formatted at 2026-08-07 00:01:06 on dist-test-slave-39l8"
server_key: "f750a6bb37e399f9d22a55210105efc8"
server_key_iv: "951e9a5a1248972de9ac20dbbf234eb4"
server_key_version: "encryptionkey@0"
I20260807 00:01:06.381099 5145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:06.383563 5168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:06.384774 5145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 00:01:06.384917 5145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "f7781d35666247ad85fea2b427743abf"
format_stamp: "Formatted at 2026-08-07 00:01:06 on dist-test-slave-39l8"
server_key: "f750a6bb37e399f9d22a55210105efc8"
server_key_iv: "951e9a5a1248972de9ac20dbbf234eb4"
server_key_version: "encryptionkey@0"
I20260807 00:01:06.385031 5145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:06.389595 5145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:06.392460 5145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:06.392652 5145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:06.399726 5145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37387
I20260807 00:01:06.399756 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37387 every 8 connection(s)
I20260807 00:01:06.400611 5145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:06.401795 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5145
I20260807 00:01:06.401939 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:06.402228 9009 external_mini_cluster.cc:1428] Setting key dd7a8c911dc9b3d3f8007f0b2b2fc5e2
I20260807 00:01:06.403714 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:06.407233 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:06.403522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34111 (local address 127.0.0.1:37387)
0807 00:01:06.403914 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:06.403930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:06.403975 (+ 45us) server_negotiation.cc:407] Connection header received
0807 00:01:06.404605 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:06.404620 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:06.404875 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:06.405144 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:06.405961 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:06.405965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:06.405977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:06.405987 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:06.406035 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:06.406053 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:06.406187 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:06.406311 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 00:01:06.406467 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260807 00:01:06.409814 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:06.409812 5240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf: Bootstrap starting.
I20260807 00:01:06.412568 5240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:06.413348 5240 log.cc:824] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:06.413779 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:06.412085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58605 (local address 127.0.0.1:37387)
0807 00:01:06.412210 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:06.412214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:06.412401 (+ 187us) server_negotiation.cc:407] Connection header received
0807 00:01:06.412468 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:06.412474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:06.412516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:06.412612 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:06.413307 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:06.413310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:06.413313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:06.413324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:06.413352 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:06.413368 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:06.413468 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:06.413592 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 00:01:06.413629 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 00:01:06.415405 5240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf: No bootstrap required, opened a new log
I20260807 00:01:06.418927 5240 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260807 00:01:06.419139 5240 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:06.419255 5240 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7781d35666247ad85fea2b427743abf, State: Initialized, Role: FOLLOWER
I20260807 00:01:06.419711 5240 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260807 00:01:06.419859 5240 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:06.419932 5240 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:06.420030 5240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:06.420890 5240 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260807 00:01:06.421252 5240 leader_election.cc:302] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7781d35666247ad85fea2b427743abf; no voters:
I20260807 00:01:06.421599 5240 leader_election.cc:288] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:06.421676 5253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:06.421913 5253 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [term 1 LEADER]: Becoming Leader. State: Replica: f7781d35666247ad85fea2b427743abf, State: Running, Role: LEADER
I20260807 00:01:06.422243 5253 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260807 00:01:06.422937 5240 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:06.424081 5254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7781d35666247ad85fea2b427743abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } } }
I20260807 00:01:06.424115 5255 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7781d35666247ad85fea2b427743abf. Latest consensus state: current_term: 1 leader_uuid: "f7781d35666247ad85fea2b427743abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7781d35666247ad85fea2b427743abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } } }
I20260807 00:01:06.424278 5255 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:06.424266 5254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:06.425197 5266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:06.427938 5266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:06.433265 5266 catalog_manager.cc:1347] Generated new cluster ID: e7d870f34b754f51ae9130b979f5acba
I20260807 00:01:06.433357 5266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:06.443284 5266 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:06.443581 5266 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:06.451654 5266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7781d35666247ad85fea2b427743abf: Generated new TSK 0
I20260807 00:01:06.452426 5266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:06.455346 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5145
2026-08-07T00:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:06 dist-test-slave-39l8 krb5kdc[5283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:06 dist-test-slave-39l8 krb5kdc[5283](info): set up 2 sockets
Aug 07 00:01:06 dist-test-slave-39l8 krb5kdc[5283](info): commencing operation
krb5kdc: starting...
W20260807 00:01:07.665395 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:07 dist-test-slave-39l8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:07.801244 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:07.917980 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:07.918334 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:07.918416 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:07.922497 5302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:07.922616 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:07.922652 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:07.922693 5302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:07.922745 5302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:07.928268 5302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:07.929514 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:07.930627 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:07.939603 5309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:07.939627 5312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:07.939603 5308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:07.939837 5302 server_base.cc:1034] running on GCE node
I20260807 00:01:07.940557 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:07.941746 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:07.942936 5302 hybrid_clock.cc:648] HybridClock initialized: now 1786060867942916 us; error 32 us; skew 500 ppm
Aug 07 00:01:07 dist-test-slave-39l8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:07.945739 5302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:07.946866 5302 webserver.cc:460] Webserver started at http://127.0.0.1:35473/ using document root <none> and password file <none>
I20260807 00:01:07.947438 5302 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:07.947500 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:07.947683 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:07.949816 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "5e113cee55114235a65e01e3986afbea"
format_stamp: "Formatted at 2026-08-07 00:01:07 on dist-test-slave-39l8"
server_key: "a1bad6c7b7fbc5ed2c63274f54960e86"
server_key_iv: "192271a19ee789c57847ae3a8cf58140"
server_key_version: "encryptionkey@0"
I20260807 00:01:07.950363 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "5e113cee55114235a65e01e3986afbea"
format_stamp: "Formatted at 2026-08-07 00:01:07 on dist-test-slave-39l8"
server_key: "a1bad6c7b7fbc5ed2c63274f54960e86"
server_key_iv: "192271a19ee789c57847ae3a8cf58140"
server_key_version: "encryptionkey@0"
I20260807 00:01:07.954267 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 00:01:07.956859 5319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:07.958206 5302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:07.958354 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "5e113cee55114235a65e01e3986afbea"
format_stamp: "Formatted at 2026-08-07 00:01:07 on dist-test-slave-39l8"
server_key: "a1bad6c7b7fbc5ed2c63274f54960e86"
server_key_iv: "192271a19ee789c57847ae3a8cf58140"
server_key_version: "encryptionkey@0"
I20260807 00:01:07.958473 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:08.013229 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:08.018236 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:08.018458 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:08.026477 5302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46475
I20260807 00:01:08.026562 5391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46475 every 8 connection(s)
I20260807 00:01:08.027536 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:08.029537 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5302
I20260807 00:01:08.029696 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:08.029981 9009 external_mini_cluster.cc:1428] Setting key 8b90fced9dd1efc706490d657ebc24ac
I20260807 00:01:08.030696 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:08.038020 5392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea: Bootstrap starting.
Aug 07 00:01:08 dist-test-slave-39l8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 00:01:08.040875 5392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:08.041879 5392 log.cc:824] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:08.044281 5392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea: No bootstrap required, opened a new log
I20260807 00:01:08.045898 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:08.031505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36793 (local address 127.0.0.1:46475)
0807 00:01:08.031952 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:08.031985 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:08.032032 (+ 47us) server_negotiation.cc:407] Connection header received
0807 00:01:08.032906 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:08.032931 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:08.033371 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:08.033757 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:08.034457 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:08.036010 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:08.036932 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:08.037620 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:08.037793 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:08.040710 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:08.040719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:08.040741 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:08.040781 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:08.042754 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:08.043195 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:08.043201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:08.043207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:08.043344 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:08.043626 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:08.043629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:08.043631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:08.044142 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:08.044337 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:08.044655 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 00:01:08.044964 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260807 00:01:08.048768 5392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } }
I20260807 00:01:08.049044 5392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:08.049180 5392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e113cee55114235a65e01e3986afbea, State: Initialized, Role: FOLLOWER
I20260807 00:01:08.049185 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:08.049677 5392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } }
I20260807 00:01:08.049808 5392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:08.049896 5392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:08.050002 5392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:08.051143 5392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } }
I20260807 00:01:08.051534 5392 leader_election.cc:302] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e113cee55114235a65e01e3986afbea; no voters:
I20260807 00:01:08.051864 5392 leader_election.cc:288] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:08.052013 5408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:08.052261 5408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [term 1 LEADER]: Becoming Leader. State: Replica: 5e113cee55114235a65e01e3986afbea, State: Running, Role: LEADER
I20260807 00:01:08.052627 5408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } }
I20260807 00:01:08.053179 5392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:08.054401 5409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e113cee55114235a65e01e3986afbea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } } }
I20260807 00:01:08.054584 5409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:08.054400 5412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e113cee55114235a65e01e3986afbea. Latest consensus state: current_term: 1 leader_uuid: "5e113cee55114235a65e01e3986afbea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e113cee55114235a65e01e3986afbea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46475 } } }
I20260807 00:01:08.054823 5412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:08.055053 5426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:08.058426 5426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:08.064518 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:08.051429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56503 (local address 127.0.0.1:46475)
0807 00:01:08.051565 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:08.051572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:08.055868 (+ 4296us) server_negotiation.cc:407] Connection header received
0807 00:01:08.055986 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:08.055989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:08.056040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:08.056135 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:08.056954 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:08.057483 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:08.058682 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:08.059073 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:08.059182 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:08.060834 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:08.060838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:08.060841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:08.060867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:08.062642 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:08.063181 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:08.063185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:08.063194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:08.063289 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:08.063558 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:08.063562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:08.063563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:08.063796 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:08.063934 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:08.064234 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 00:01:08.064357 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 00:01:08.065500 5426 catalog_manager.cc:1347] Generated new cluster ID: 30d02aa79b714c52b8fc763ac8aacdb8
I20260807 00:01:08.065616 5426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:08.072069 5426 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:08.073066 5426 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:08.083209 5426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e113cee55114235a65e01e3986afbea: Generated new TSK 0
I20260807 00:01:08.084115 5426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:08.095669 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5302
2026-08-07T00:01:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1637 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:08 dist-test-slave-39l8 krb5kdc[5442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:08 dist-test-slave-39l8 krb5kdc[5442](info): set up 2 sockets
Aug 07 00:01:08 dist-test-slave-39l8 krb5kdc[5442](info): commencing operation
krb5kdc: starting...
W20260807 00:01:09.302505 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:09 dist-test-slave-39l8 krb5kdc[5442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:09.423923 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:09.536208 5456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:09.536461 5456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:09.536501 5456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:09.540858 5456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:09.540913 5456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:09.540933 5456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:09.540949 5456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:09.540966 5456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:09.546118 5456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:09.547156 5456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:09.548106 5456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:09.556943 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:09.556955 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:09.556955 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:09.557219 5456 server_base.cc:1034] running on GCE node
I20260807 00:01:09.557844 5456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:09.559064 5456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:09.560258 5456 hybrid_clock.cc:648] HybridClock initialized: now 1786060869560233 us; error 35 us; skew 500 ppm
Aug 07 00:01:09 dist-test-slave-39l8 krb5kdc[5442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:09.563066 5456 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:09.563926 5456 webserver.cc:460] Webserver started at http://127.0.0.1:46787/ using document root <none> and password file <none>
I20260807 00:01:09.564487 5456 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:09.564581 5456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:09.564814 5456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:09.566993 5456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "736f0a93c5e9469092cce25ca8a491ba"
format_stamp: "Formatted at 2026-08-07 00:01:09 on dist-test-slave-39l8"
server_key: "17f8c5dc3e6a3115be6ea1584d34ffed"
server_key_iv: "aa8297209d3dc9a80ff7a2658098eb32"
server_key_version: "encryptionkey@0"
I20260807 00:01:09.567562 5456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "736f0a93c5e9469092cce25ca8a491ba"
format_stamp: "Formatted at 2026-08-07 00:01:09 on dist-test-slave-39l8"
server_key: "17f8c5dc3e6a3115be6ea1584d34ffed"
server_key_iv: "aa8297209d3dc9a80ff7a2658098eb32"
server_key_version: "encryptionkey@0"
I20260807 00:01:09.571224 5456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 00:01:09.573590 5475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:09.574841 5456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:09.574983 5456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "736f0a93c5e9469092cce25ca8a491ba"
format_stamp: "Formatted at 2026-08-07 00:01:09 on dist-test-slave-39l8"
server_key: "17f8c5dc3e6a3115be6ea1584d34ffed"
server_key_iv: "aa8297209d3dc9a80ff7a2658098eb32"
server_key_version: "encryptionkey@0"
I20260807 00:01:09.575124 5456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:09.588905 5456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:09.591789 5456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:09.592013 5456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:09.599252 5456 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44217
I20260807 00:01:09.599282 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44217 every 8 connection(s)
I20260807 00:01:09.600188 5456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:09.602962 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:09.608588 5550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba: Bootstrap starting.
I20260807 00:01:09.610494 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5456
I20260807 00:01:09.610652 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:09.610993 9009 external_mini_cluster.cc:1428] Setting key 3dd2eff614401b3f94448b72671ed5c7
I20260807 00:01:09.611369 5550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:09.612212 5550 log.cc:824] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:09.614198 5550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba: No bootstrap required, opened a new log
I20260807 00:01:09.617849 5550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } }
I20260807 00:01:09.618119 5550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:09.618204 5550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736f0a93c5e9469092cce25ca8a491ba, State: Initialized, Role: FOLLOWER
I20260807 00:01:09.618750 5550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } }
I20260807 00:01:09.618919 5550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:09.618988 5550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:09.619103 5550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:09.620208 5550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } }
I20260807 00:01:09.620550 5550 leader_election.cc:302] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736f0a93c5e9469092cce25ca8a491ba; no voters:
I20260807 00:01:09.620852 5550 leader_election.cc:288] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:09.620898 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:09.612285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33501 (local address 127.0.0.1:44217)
0807 00:01:09.612885 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:09.612907 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:09.612955 (+ 48us) server_negotiation.cc:407] Connection header received
0807 00:01:09.613656 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:09.613673 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:09.614006 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:09.614338 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:09.614831 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:09.615781 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:09.617039 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:09.617522 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:09.617700 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:09.618664 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:09.618671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:09.618690 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:09.619331 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:09.619397 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:09.619424 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:09.619515 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:09.619699 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 00:01:09.619956 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":139,"threads_started":1}
I20260807 00:01:09.620983 5557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:09.621323 5557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [term 1 LEADER]: Becoming Leader. State: Replica: 736f0a93c5e9469092cce25ca8a491ba, State: Running, Role: LEADER
I20260807 00:01:09.621701 5557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } }
I20260807 00:01:09.622179 5550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:09.623100 5557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736f0a93c5e9469092cce25ca8a491ba. Latest consensus state: current_term: 1 leader_uuid: "736f0a93c5e9469092cce25ca8a491ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } } }
I20260807 00:01:09.623298 5557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:09.623772 5566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:09.624284 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:09.625229 5559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736f0a93c5e9469092cce25ca8a491ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f0a93c5e9469092cce25ca8a491ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44217 } } }
I20260807 00:01:09.625337 5559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:09.627761 5566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:09.634392 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:09.627161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37417 (local address 127.0.0.1:44217)
0807 00:01:09.627281 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:09.627287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:09.629968 (+ 2681us) server_negotiation.cc:407] Connection header received
0807 00:01:09.630098 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:09.630101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:09.630151 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:09.630243 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:09.630571 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:09.631422 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:09.632416 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:09.632719 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:09.632850 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:09.633883 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:09.633887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:09.633890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:09.633903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:09.633934 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:09.633949 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:09.634032 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:09.634187 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 00:01:09.634248 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 00:01:09.634800 5566 catalog_manager.cc:1347] Generated new cluster ID: 890871ee6a1849b0a5c261c022566bbb
I20260807 00:01:09.634861 5566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:09.649751 5566 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:09.650817 5566 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:09.668161 5566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736f0a93c5e9469092cce25ca8a491ba: Generated new TSK 0
I20260807 00:01:09.668941 5566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:09.673400 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5456
2026-08-07T00:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:09 dist-test-slave-39l8 krb5kdc[5591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:09 dist-test-slave-39l8 krb5kdc[5591](info): set up 2 sockets
Aug 07 00:01:09 dist-test-slave-39l8 krb5kdc[5591](info): commencing operation
krb5kdc: starting...
W20260807 00:01:10.891134 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:10 dist-test-slave-39l8 krb5kdc[5591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:11.014066 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:11.128069 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:11.128374 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:11.128453 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:11.132428 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:11.132524 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:11.132565 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:11.132608 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:11.132645 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:11.137861 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:11.138957 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:11.139981 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:11.148550 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:11.148554 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:11.148662 5610 server_base.cc:1034] running on GCE node
W20260807 00:01:11.148550 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:11.149415 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:11.150496 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:11.151695 5610 hybrid_clock.cc:648] HybridClock initialized: now 1786060871151662 us; error 43 us; skew 500 ppm
Aug 07 00:01:11 dist-test-slave-39l8 krb5kdc[5591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:11.158957 5610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:11.159873 5610 webserver.cc:460] Webserver started at http://127.0.0.1:35811/ using document root <none> and password file <none>
I20260807 00:01:11.160455 5610 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:11.160548 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:11.160797 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:11.163022 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "eafe216c2aee4af48a9b75f503ed3ca7"
format_stamp: "Formatted at 2026-08-07 00:01:11 on dist-test-slave-39l8"
server_key: "4471b83ba20298cf0315778d704efff0"
server_key_iv: "73e78e60f458f2218fa20c149de2bdfc"
server_key_version: "encryptionkey@0"
I20260807 00:01:11.163609 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "eafe216c2aee4af48a9b75f503ed3ca7"
format_stamp: "Formatted at 2026-08-07 00:01:11 on dist-test-slave-39l8"
server_key: "4471b83ba20298cf0315778d704efff0"
server_key_iv: "73e78e60f458f2218fa20c149de2bdfc"
server_key_version: "encryptionkey@0"
I20260807 00:01:11.167240 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 00:01:11.169601 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:11.170857 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:11.171000 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "eafe216c2aee4af48a9b75f503ed3ca7"
format_stamp: "Formatted at 2026-08-07 00:01:11 on dist-test-slave-39l8"
server_key: "4471b83ba20298cf0315778d704efff0"
server_key_iv: "73e78e60f458f2218fa20c149de2bdfc"
server_key_version: "encryptionkey@0"
I20260807 00:01:11.171113 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:11.175603 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:11.178402 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:11.178601 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:11.185976 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36999
I20260807 00:01:11.186000 5704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36999 every 8 connection(s)
I20260807 00:01:11.186997 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:11.189791 5705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:11.191406 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5610
I20260807 00:01:11.191543 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:11.191821 9009 external_mini_cluster.cc:1428] Setting key 6e5b92118828b2e5293f5da75a64d5da
I20260807 00:01:11.196089 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7: Bootstrap starting.
I20260807 00:01:11.196970 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:11.193254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46889 (local address 127.0.0.1:36999)
0807 00:01:11.193696 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:11.193710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:11.193745 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:11.194360 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:11.194375 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:11.194635 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:11.195022 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:11.195824 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:11.195828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:11.195840 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:11.195850 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:11.195901 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:11.195919 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:11.196069 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:11.196136 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 00:01:11.196300 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260807 00:01:11.198551 5705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:11.199469 5705 log.cc:824] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:11.199905 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:11.201537 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7: No bootstrap required, opened a new log
I20260807 00:01:11.203737 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:11.201987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37189 (local address 127.0.0.1:36999)
0807 00:01:11.202112 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:11.202116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:11.202329 (+ 213us) server_negotiation.cc:407] Connection header received
0807 00:01:11.202449 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:11.202456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:11.202493 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:11.202578 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:11.203346 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:11.203350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:11.203352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:11.203364 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:11.203396 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:11.203407 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:11.203496 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:11.203571 (+ 75us) server_negotiation.cc:299] Negotiation successful
0807 00:01:11.203611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 00:01:11.205739 5705 raft_consensus.cc:348] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260807 00:01:11.205955 5705 raft_consensus.cc:374] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:11.206030 5705 raft_consensus.cc:729] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafe216c2aee4af48a9b75f503ed3ca7, State: Initialized, Role: FOLLOWER
I20260807 00:01:11.206449 5705 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260807 00:01:11.206596 5705 raft_consensus.cc:388] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:11.206666 5705 raft_consensus.cc:482] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:11.206791 5705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:11.207684 5705 raft_consensus.cc:504] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260807 00:01:11.208024 5705 leader_election.cc:302] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafe216c2aee4af48a9b75f503ed3ca7; no voters:
I20260807 00:01:11.208384 5705 leader_election.cc:288] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:11.208498 5719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:11.208720 5719 raft_consensus.cc:686] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [term 1 LEADER]: Becoming Leader. State: Replica: eafe216c2aee4af48a9b75f503ed3ca7, State: Running, Role: LEADER
I20260807 00:01:11.209075 5719 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260807 00:01:11.209623 5705 sys_catalog.cc:564] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:11.210736 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260807 00:01:11.210873 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:11.211279 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafe216c2aee4af48a9b75f503ed3ca7. Latest consensus state: current_term: 1 leader_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafe216c2aee4af48a9b75f503ed3ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260807 00:01:11.211374 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:11.211434 5733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:11.214043 5733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:11.220110 5733 catalog_manager.cc:1347] Generated new cluster ID: f0616ab0b71c43cf80b9131f656b0605
I20260807 00:01:11.220177 5733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:11.235301 5733 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:11.235610 5733 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:11.243260 5733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eafe216c2aee4af48a9b75f503ed3ca7: Generated new TSK 0
I20260807 00:01:11.244006 5733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:11.248188 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5610
2026-08-07T00:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1573 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:11 dist-test-slave-39l8 krb5kdc[5748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:11 dist-test-slave-39l8 krb5kdc[5748](info): set up 2 sockets
Aug 07 00:01:11 dist-test-slave-39l8 krb5kdc[5748](info): commencing operation
krb5kdc: starting...
W20260807 00:01:12.455298 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:12.579933 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:12.691324 5765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:12.691567 5765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:12.691607 5765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:12.695890 5765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:12.695971 5765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:12.696028 5765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:12.696064 5765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:12.696094 5765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:12.701930 5765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:12.703063 5765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:12.704066 5765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:12.712626 5770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:12.712647 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:12.712713 5765 server_base.cc:1034] running on GCE node
W20260807 00:01:12.712626 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:12.713496 5765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:12.714746 5765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:12.715931 5765 hybrid_clock.cc:648] HybridClock initialized: now 1786060872715911 us; error 32 us; skew 500 ppm
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:12.718745 5765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:12.719559 5765 webserver.cc:460] Webserver started at http://127.0.0.1:33321/ using document root <none> and password file <none>
I20260807 00:01:12.720068 5765 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:12.720139 5765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:12.720299 5765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:12.722196 5765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "3b67b61d276647b485c44b61a6aebaeb"
format_stamp: "Formatted at 2026-08-07 00:01:12 on dist-test-slave-39l8"
server_key: "f748a84432e6da11bbc9a7d197a3009b"
server_key_iv: "ed9e10467ce68d2b5c9f2553b173aa67"
server_key_version: "encryptionkey@0"
I20260807 00:01:12.722697 5765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "3b67b61d276647b485c44b61a6aebaeb"
format_stamp: "Formatted at 2026-08-07 00:01:12 on dist-test-slave-39l8"
server_key: "f748a84432e6da11bbc9a7d197a3009b"
server_key_iv: "ed9e10467ce68d2b5c9f2553b173aa67"
server_key_version: "encryptionkey@0"
I20260807 00:01:12.726390 5765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 00:01:12.728752 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:12.729919 5765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:12.730063 5765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "3b67b61d276647b485c44b61a6aebaeb"
format_stamp: "Formatted at 2026-08-07 00:01:12 on dist-test-slave-39l8"
server_key: "f748a84432e6da11bbc9a7d197a3009b"
server_key_iv: "ed9e10467ce68d2b5c9f2553b173aa67"
server_key_version: "encryptionkey@0"
I20260807 00:01:12.730182 5765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:12.740571 5765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:12.743749 5765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:12.743963 5765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:12.751107 5765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34861
I20260807 00:01:12.751118 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34861 every 8 connection(s)
I20260807 00:01:12.752039 5765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:12.754841 5855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:12.756808 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5765
I20260807 00:01:12.756965 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:12.757237 9009 external_mini_cluster.cc:1428] Setting key dd62826e18ccf03b91e38dfbbd892ab1
I20260807 00:01:12.761291 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: Bootstrap starting.
I20260807 00:01:12.763926 5855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:12.764647 5855 log.cc:824] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:12.766534 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: No bootstrap required, opened a new log
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 00:01:12.771034 5855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } }
I20260807 00:01:12.771368 5855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:12.771466 5855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b67b61d276647b485c44b61a6aebaeb, State: Initialized, Role: FOLLOWER
I20260807 00:01:12.771924 5855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } }
I20260807 00:01:12.772091 5855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:12.772181 5855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:12.772313 5855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:12.772394 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:12.758651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:54543 (local address 127.0.0.1:34861)
0807 00:01:12.759118 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:12.759138 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:12.759180 (+ 42us) server_negotiation.cc:407] Connection header received
0807 00:01:12.760171 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:12.760207 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:12.760615 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:12.760961 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:12.761699 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:12.763199 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:12.764052 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:12.764517 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:12.764724 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:12.767394 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:12.767399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:12.767411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:12.767437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:12.769111 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:12.769689 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:12.769693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:12.769698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:12.769809 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:12.770177 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:12.770179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:12.770181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:12.770559 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:12.770813 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:12.771122 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 00:01:12.771378 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":109,"threads_started":1}
I20260807 00:01:12.773550 5855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } }
I20260807 00:01:12.773945 5855 leader_election.cc:302] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b67b61d276647b485c44b61a6aebaeb; no voters:
I20260807 00:01:12.774314 5855 leader_election.cc:288] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:12.774441 5865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:12.774752 5865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [term 1 LEADER]: Becoming Leader. State: Replica: 3b67b61d276647b485c44b61a6aebaeb, State: Running, Role: LEADER
I20260807 00:01:12.775146 5865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } }
I20260807 00:01:12.775693 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:12.775722 5855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:12.776422 5865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b67b61d276647b485c44b61a6aebaeb. Latest consensus state: current_term: 1 leader_uuid: "3b67b61d276647b485c44b61a6aebaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } } }
I20260807 00:01:12.776628 5865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:12.776943 5866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b67b61d276647b485c44b61a6aebaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b67b61d276647b485c44b61a6aebaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34861 } } }
I20260807 00:01:12.777045 5866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb [sys.catalog]: This master's current role is: LEADER
W20260807 00:01:12.781469 5892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:12.781565 5892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:12.781646 5881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:12.782500 5881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:12.788213 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:12.778378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:56187 (local address 127.0.0.1:34861)
0807 00:01:12.778519 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:12.778527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:12.778653 (+ 126us) server_negotiation.cc:407] Connection header received
0807 00:01:12.778886 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:12.778889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:12.778943 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:12.779000 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:12.779743 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:12.780527 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:12.781702 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:12.782034 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:12.782147 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:12.783926 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:12.783932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:12.783937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:12.783964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:12.785741 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:12.786331 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:12.786335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:12.786337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:12.786458 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:12.786909 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:12.786915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:12.786917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:12.787145 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:12.787832 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:12.787939 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 00:01:12.788033 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:01:12.788462 5881 catalog_manager.cc:1347] Generated new cluster ID: 0e27d5a985ce41c99354946c5766692d
I20260807 00:01:12.788518 5881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:12.797868 5881 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:12.799001 5881 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:12.807914 5881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b67b61d276647b485c44b61a6aebaeb: Generated new TSK 0
I20260807 00:01:12.808630 5881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:12.815629 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5765
W20260807 00:01:12.831116 5745 connection.cc:535] client connection to 127.0.0.1:34861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34861): Connection reset by peer (error 104)
2026-08-07T00:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1571 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5903](info): set up 2 sockets
Aug 07 00:01:12 dist-test-slave-39l8 krb5kdc[5903](info): commencing operation
krb5kdc: starting...
W20260807 00:01:14.011032 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:14 dist-test-slave-39l8 krb5kdc[5903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:14.142505 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:14.255478 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:14.255806 5918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:14.255851 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:14.259896 5918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:14.259955 5918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:14.259974 5918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:14.260020 5918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:14.260057 5918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:14.265333 5918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:14.266412 5918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:14.267470 5918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:14.275760 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:14.275760 5927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:14.275921 5918 server_base.cc:1034] running on GCE node
W20260807 00:01:14.275840 5925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:14.276528 5918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:14.277590 5918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:14.278795 5918 hybrid_clock.cc:648] HybridClock initialized: now 1786060874278729 us; error 76 us; skew 500 ppm
Aug 07 00:01:14 dist-test-slave-39l8 krb5kdc[5903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:14.281647 5918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:14.282438 5918 webserver.cc:460] Webserver started at http://127.0.0.1:41253/ using document root <none> and password file <none>
I20260807 00:01:14.283031 5918 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:14.283097 5918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:14.283278 5918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:14.285189 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "301125da7d214e5cb94d4979ac216aff"
format_stamp: "Formatted at 2026-08-07 00:01:14 on dist-test-slave-39l8"
server_key: "96b65b88c7dc9afa7c8aa0451a61a9c3"
server_key_iv: "3b09ef8cfc148b7cb7e846cdbec1395f"
server_key_version: "encryptionkey@0"
I20260807 00:01:14.285717 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "301125da7d214e5cb94d4979ac216aff"
format_stamp: "Formatted at 2026-08-07 00:01:14 on dist-test-slave-39l8"
server_key: "96b65b88c7dc9afa7c8aa0451a61a9c3"
server_key_iv: "3b09ef8cfc148b7cb7e846cdbec1395f"
server_key_version: "encryptionkey@0"
I20260807 00:01:14.289338 5918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 00:01:14.291725 5939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:14.292914 5918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:14.293016 5918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "301125da7d214e5cb94d4979ac216aff"
format_stamp: "Formatted at 2026-08-07 00:01:14 on dist-test-slave-39l8"
server_key: "96b65b88c7dc9afa7c8aa0451a61a9c3"
server_key_iv: "3b09ef8cfc148b7cb7e846cdbec1395f"
server_key_version: "encryptionkey@0"
I20260807 00:01:14.293102 5918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:14.305137 5918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:14.308300 5918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:14.308449 5918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:14.315429 5918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39667
I20260807 00:01:14.315447 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39667 every 8 connection(s)
I20260807 00:01:14.316442 5918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:14.319236 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:14.319432 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 5918
I20260807 00:01:14.319591 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:14.319882 9009 external_mini_cluster.cc:1428] Setting key bc9c71a2edf6b0d056a08a6f304b83e9
I20260807 00:01:14.325603 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff: Bootstrap starting.
I20260807 00:01:14.327857 6011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:14.328704 6011 log.cc:824] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:14.329026 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:14.321292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:50219 (local address 127.0.0.1:39667)
0807 00:01:14.321734 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:14.321748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:14.321783 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:14.322493 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:14.322511 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:14.322923 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:14.323257 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:14.323655 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:14.324710 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:14.325635 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:14.326162 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:14.326360 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:14.327338 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:14.327343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:14.327355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:14.327743 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:14.327798 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:14.327821 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:14.327969 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:14.328148 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 00:01:14.328289 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260807 00:01:14.330765 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff: No bootstrap required, opened a new log
I20260807 00:01:14.332042 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:14.335399 6011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } }
I20260807 00:01:14.335608 6011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:14.335665 6011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301125da7d214e5cb94d4979ac216aff, State: Initialized, Role: FOLLOWER
I20260807 00:01:14.336141 6011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } }
I20260807 00:01:14.336309 6011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:14.336377 6011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:14.336469 6011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:14.337544 6011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } }
I20260807 00:01:14.337939 6011 leader_election.cc:302] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 301125da7d214e5cb94d4979ac216aff; no voters:
I20260807 00:01:14.338315 6011 leader_election.cc:288] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:14.338387 6023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:14.338608 6023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [term 1 LEADER]: Becoming Leader. State: Replica: 301125da7d214e5cb94d4979ac216aff, State: Running, Role: LEADER
I20260807 00:01:14.338774 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:14.334385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:54345 (local address 127.0.0.1:39667)
0807 00:01:14.334497 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:14.334503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:14.334675 (+ 172us) server_negotiation.cc:407] Connection header received
0807 00:01:14.334813 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:14.334815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:14.334855 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:14.334916 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:14.335220 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:14.335763 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:14.336878 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:14.337136 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:14.337220 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:14.338172 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:14.338176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:14.338179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:14.338195 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:14.338218 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:14.338228 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:14.338332 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:14.338529 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 00:01:14.338563 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 00:01:14.338984 6023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } }
I20260807 00:01:14.339637 6011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:14.340713 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "301125da7d214e5cb94d4979ac216aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } } }
I20260807 00:01:14.340873 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:14.341080 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 301125da7d214e5cb94d4979ac216aff. Latest consensus state: current_term: 1 leader_uuid: "301125da7d214e5cb94d4979ac216aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301125da7d214e5cb94d4979ac216aff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39667 } } }
I20260807 00:01:14.341233 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:14.341245 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:14.344486 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:14.350160 6034 catalog_manager.cc:1347] Generated new cluster ID: a797e67bdfb8411286cda6f1e68aceaa
I20260807 00:01:14.350265 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:14.361704 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:14.362557 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:14.368607 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 301125da7d214e5cb94d4979ac216aff: Generated new TSK 0
I20260807 00:01:14.369204 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:14.378333 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 5918
W20260807 00:01:14.388432 5900 connection.cc:535] client connection to 127.0.0.1:39667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39667): Connection reset by peer (error 104)
2026-08-07T00:01:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1557 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:14 dist-test-slave-39l8 krb5kdc[6052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:14 dist-test-slave-39l8 krb5kdc[6052](info): set up 2 sockets
Aug 07 00:01:14 dist-test-slave-39l8 krb5kdc[6052](info): commencing operation
krb5kdc: starting...
W20260807 00:01:15.563441 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:15 dist-test-slave-39l8 krb5kdc[6052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:15.693486 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:15.808661 6067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:15.808977 6067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:15.809072 6067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:15.813153 6067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:15.813244 6067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:15.813282 6067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:15.813344 6067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:15.813382 6067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:15.818634 6067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:15.819787 6067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:15.820855 6067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:15.829392 6074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:15.829418 6075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:15.829392 6077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:15.829676 6067 server_base.cc:1034] running on GCE node
I20260807 00:01:15.830277 6067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:15.831424 6067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:15.832587 6067 hybrid_clock.cc:648] HybridClock initialized: now 1786060875832567 us; error 30 us; skew 500 ppm
Aug 07 00:01:15 dist-test-slave-39l8 krb5kdc[6052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:15.835314 6067 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:15.836128 6067 webserver.cc:460] Webserver started at http://127.0.0.1:37405/ using document root <none> and password file <none>
I20260807 00:01:15.836633 6067 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:15.836697 6067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:15.836860 6067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:15.838860 6067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "bcaeada8fdc34decb37f148a4c499926"
format_stamp: "Formatted at 2026-08-07 00:01:15 on dist-test-slave-39l8"
server_key: "7ed6a69040ca70c1f5b7754427b2a1ba"
server_key_iv: "a134c7a4109a4cebd5e8e22c01a67258"
server_key_version: "encryptionkey@0"
I20260807 00:01:15.839382 6067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "bcaeada8fdc34decb37f148a4c499926"
format_stamp: "Formatted at 2026-08-07 00:01:15 on dist-test-slave-39l8"
server_key: "7ed6a69040ca70c1f5b7754427b2a1ba"
server_key_iv: "a134c7a4109a4cebd5e8e22c01a67258"
server_key_version: "encryptionkey@0"
I20260807 00:01:15.843451 6067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 00:01:15.845809 6087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:15.847122 6067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 00:01:15.847258 6067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "bcaeada8fdc34decb37f148a4c499926"
format_stamp: "Formatted at 2026-08-07 00:01:15 on dist-test-slave-39l8"
server_key: "7ed6a69040ca70c1f5b7754427b2a1ba"
server_key_iv: "a134c7a4109a4cebd5e8e22c01a67258"
server_key_version: "encryptionkey@0"
I20260807 00:01:15.847365 6067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:15.852028 6067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:15.855026 6067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:15.855225 6067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:15.862259 6067 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42445
I20260807 00:01:15.862358 6154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42445 every 8 connection(s)
I20260807 00:01:15.863305 6067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:15.866111 6157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:15.870630 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6067
I20260807 00:01:15.870797 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:15.871068 9009 external_mini_cluster.cc:1428] Setting key 54fc8cba6ae05aebdf9d5f6e0d988b90
I20260807 00:01:15.872035 6157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: Bootstrap starting.
I20260807 00:01:15.874861 6157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:15.875797 6157 log.cc:824] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:15.876616 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:15.872474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:48105 (local address 127.0.0.1:42445)
0807 00:01:15.872902 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:15.872920 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:15.872962 (+ 42us) server_negotiation.cc:407] Connection header received
0807 00:01:15.873828 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:15.873852 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:15.874197 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:15.874544 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:15.875275 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:15.875281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:15.875295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:15.875306 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:15.875357 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:15.875381 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:15.875485 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:15.875616 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 00:01:15.875775 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260807 00:01:15.877728 6157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: No bootstrap required, opened a new log
I20260807 00:01:15.879297 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:15.882332 6157 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } }
I20260807 00:01:15.882623 6157 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:15.882688 6157 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcaeada8fdc34decb37f148a4c499926, State: Initialized, Role: FOLLOWER
I20260807 00:01:15.883219 6157 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } }
I20260807 00:01:15.883399 6157 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:15.883495 6157 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:15.883610 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:15.881651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:56091 (local address 127.0.0.1:42445)
0807 00:01:15.881857 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:15.881861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:15.882104 (+ 243us) server_negotiation.cc:407] Connection header received
0807 00:01:15.882213 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:15.882216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:15.882244 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:15.882307 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:15.883095 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:15.883098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:15.883100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:15.883110 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:15.883134 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:15.883144 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:15.883216 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:15.883335 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 00:01:15.883395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 00:01:15.883626 6157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:15.884825 6157 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } }
I20260807 00:01:15.885172 6157 leader_election.cc:302] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcaeada8fdc34decb37f148a4c499926; no voters:
I20260807 00:01:15.885475 6157 leader_election.cc:288] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:15.885576 6171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:15.885812 6171 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [term 1 LEADER]: Becoming Leader. State: Replica: bcaeada8fdc34decb37f148a4c499926, State: Running, Role: LEADER
I20260807 00:01:15.886142 6171 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } }
I20260807 00:01:15.886691 6157 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:15.887964 6173 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcaeada8fdc34decb37f148a4c499926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } } }
I20260807 00:01:15.888123 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcaeada8fdc34decb37f148a4c499926. Latest consensus state: current_term: 1 leader_uuid: "bcaeada8fdc34decb37f148a4c499926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaeada8fdc34decb37f148a4c499926" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42445 } } }
I20260807 00:01:15.888208 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:15.888106 6173 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926 [sys.catalog]: This master's current role is: LEADER
W20260807 00:01:15.891763 6193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:15.891855 6193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:15.891940 6187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:15.892773 6187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:15.897769 6187 catalog_manager.cc:1347] Generated new cluster ID: acdcbe4861ca4ec3ac83d99d508d20e0
I20260807 00:01:15.897835 6187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:15.908912 6187 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:15.909225 6187 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:15.918969 6187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcaeada8fdc34decb37f148a4c499926: Generated new TSK 0
I20260807 00:01:15.919608 6187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:15.927137 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6067
2026-08-07T00:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:15 dist-test-slave-39l8 krb5kdc[6200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:15 dist-test-slave-39l8 krb5kdc[6200](info): set up 2 sockets
Aug 07 00:01:15 dist-test-slave-39l8 krb5kdc[6200](info): commencing operation
krb5kdc: starting...
W20260807 00:01:17.121354 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:17.239605 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:17.350831 6217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:17.351109 6217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:17.351208 6217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:17.355350 6217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:17.355424 6217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:17.355463 6217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:17.355515 6217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:17.355553 6217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:17.360594 6217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46873
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:17.361605 6217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:17.362587 6217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:17.371091 6226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:17.371073 6228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:17.371232 6217 server_base.cc:1034] running on GCE node
W20260807 00:01:17.371096 6224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:17.371815 6217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:17.372920 6217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:17.374101 6217 hybrid_clock.cc:648] HybridClock initialized: now 1786060877374072 us; error 26 us; skew 500 ppm
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:17.380584 6217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:17.381510 6217 webserver.cc:460] Webserver started at http://127.0.0.1:34531/ using document root <none> and password file <none>
I20260807 00:01:17.382073 6217 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:17.382174 6217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:17.382408 6217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:17.384469 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "7a679eb9f9ce41fbb53045ddd60c94df"
format_stamp: "Formatted at 2026-08-07 00:01:17 on dist-test-slave-39l8"
server_key: "a8cf1624e7825288c0fa5d2e33a422c2"
server_key_iv: "ed87cebc6d72f8f9a4d10e0eaff5ac27"
server_key_version: "encryptionkey@0"
I20260807 00:01:17.385033 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "7a679eb9f9ce41fbb53045ddd60c94df"
format_stamp: "Formatted at 2026-08-07 00:01:17 on dist-test-slave-39l8"
server_key: "a8cf1624e7825288c0fa5d2e33a422c2"
server_key_iv: "ed87cebc6d72f8f9a4d10e0eaff5ac27"
server_key_version: "encryptionkey@0"
I20260807 00:01:17.388808 6217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:17.391163 6237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:17.392315 6217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:17.392460 6217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "7a679eb9f9ce41fbb53045ddd60c94df"
format_stamp: "Formatted at 2026-08-07 00:01:17 on dist-test-slave-39l8"
server_key: "a8cf1624e7825288c0fa5d2e33a422c2"
server_key_iv: "ed87cebc6d72f8f9a4d10e0eaff5ac27"
server_key_version: "encryptionkey@0"
I20260807 00:01:17.392570 6217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:17.420802 6217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:17.423657 6217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:17.423867 6217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:17.431460 6217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46873
I20260807 00:01:17.431506 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46873 every 8 connection(s)
I20260807 00:01:17.432449 6217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:17.435209 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:17.436168 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6217
I20260807 00:01:17.436313 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:17.436617 9009 external_mini_cluster.cc:1428] Setting key 82e53c0ecda878a2ead07704198e08e8
I20260807 00:01:17.441900 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df: Bootstrap starting.
I20260807 00:01:17.444516 6313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:17.445266 6313 log.cc:824] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df: Log is configured to *not* fsync() on all Append() calls
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 00:01:17.447578 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df: No bootstrap required, opened a new log
I20260807 00:01:17.452330 6313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260807 00:01:17.452590 6313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:17.452620 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:17.438099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:53597 (local address 127.0.0.1:46873)
0807 00:01:17.438564 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:17.438582 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:17.438624 (+ 42us) server_negotiation.cc:407] Connection header received
0807 00:01:17.439447 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:17.439470 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:17.439890 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:17.440262 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:17.441012 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:17.442437 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:17.443465 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:17.444123 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:17.444283 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:17.447230 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:17.447236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:17.447251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:17.447281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:17.449286 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:17.449802 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:17.449808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:17.449814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:17.449969 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:17.450315 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:17.450319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:17.450321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:17.450835 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:17.451051 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:17.451427 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 00:01:17.451652 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":107,"threads_started":1}
I20260807 00:01:17.452651 6313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a679eb9f9ce41fbb53045ddd60c94df, State: Initialized, Role: FOLLOWER
I20260807 00:01:17.453128 6313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260807 00:01:17.453263 6313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:17.453343 6313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:17.453445 6313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:17.454530 6313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260807 00:01:17.454936 6313 leader_election.cc:302] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a679eb9f9ce41fbb53045ddd60c94df; no voters:
I20260807 00:01:17.455330 6313 leader_election.cc:288] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:17.455407 6319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:17.455629 6319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [term 1 LEADER]: Becoming Leader. State: Replica: 7a679eb9f9ce41fbb53045ddd60c94df, State: Running, Role: LEADER
I20260807 00:01:17.456007 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:17.455991 6319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260807 00:01:17.456581 6313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:17.457612 6320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } } }
I20260807 00:01:17.457741 6320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:17.458050 6321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a679eb9f9ce41fbb53045ddd60c94df. Latest consensus state: current_term: 1 leader_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a679eb9f9ce41fbb53045ddd60c94df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } } }
I20260807 00:01:17.458217 6321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:17.458154 6335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:17.461673 6335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:17.466954 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:17.458441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:37645 (local address 127.0.0.1:46873)
0807 00:01:17.458543 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:17.458549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:17.458565 (+ 16us) server_negotiation.cc:407] Connection header received
0807 00:01:17.458606 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:17.458611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:17.458658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:17.458823 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:17.459599 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:17.460345 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:17.461552 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:17.461833 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:17.461951 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:17.463469 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:17.463472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:17.463475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:17.463498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:17.464938 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:17.465496 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:17.465500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:17.465501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:17.465582 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:17.465961 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:17.465965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:17.465966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:17.466154 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:17.466290 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:17.466674 (+ 384us) server_negotiation.cc:299] Negotiation successful
0807 00:01:17.466807 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 00:01:17.468495 6335 catalog_manager.cc:1347] Generated new cluster ID: 7208f3b49ac24cc1b19318bd52f4eca4
I20260807 00:01:17.468595 6335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:17.486224 6335 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:17.487147 6335 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:17.502525 6335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a679eb9f9ce41fbb53045ddd60c94df: Generated new TSK 0
I20260807 00:01:17.503324 6335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:17.515658 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6217
W20260807 00:01:17.529361 6196 connection.cc:535] client connection to 127.0.0.1:46873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46873): Connection reset by peer (error 104)
2026-08-07T00:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1589 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6353](info): set up 2 sockets
Aug 07 00:01:17 dist-test-slave-39l8 krb5kdc[6353](info): commencing operation
krb5kdc: starting...
W20260807 00:01:18.977648 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.434s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:19 dist-test-slave-39l8 krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:19.171813 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:19.380523 6374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:19.380970 6374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:19.381145 6374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:19.386328 6374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:19.386449 6374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:19.386540 6374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:19.386611 6374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:19.386668 6374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:19.393550 6374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:19.395103 6374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:19.396597 6374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:19.408741 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:19.408667 6381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:19.408667 6383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:19.410010 6374 server_base.cc:1034] running on GCE node
I20260807 00:01:19.410859 6374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:19.412487 6374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:19.413690 6374 hybrid_clock.cc:648] HybridClock initialized: now 1786060879413640 us; error 61 us; skew 500 ppm
Aug 07 00:01:19 dist-test-slave-39l8 krb5kdc[6353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:19.417778 6374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:19.419261 6374 webserver.cc:460] Webserver started at http://127.0.0.1:33411/ using document root <none> and password file <none>
I20260807 00:01:19.420226 6374 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:19.420365 6374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:19.420718 6374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:19.424255 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "0b2d33bf94f04a9da715cbe9500594db"
format_stamp: "Formatted at 2026-08-07 00:01:19 on dist-test-slave-39l8"
server_key: "67fc3969840d66af4f043fd29e93fc6e"
server_key_iv: "432200e89f73354ae8862db4bbc52195"
server_key_version: "encryptionkey@0"
I20260807 00:01:19.425297 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "0b2d33bf94f04a9da715cbe9500594db"
format_stamp: "Formatted at 2026-08-07 00:01:19 on dist-test-slave-39l8"
server_key: "67fc3969840d66af4f043fd29e93fc6e"
server_key_iv: "432200e89f73354ae8862db4bbc52195"
server_key_version: "encryptionkey@0"
I20260807 00:01:19.431562 6374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 00:01:19.434947 6392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:19.436766 6374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 00:01:19.436965 6374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "0b2d33bf94f04a9da715cbe9500594db"
format_stamp: "Formatted at 2026-08-07 00:01:19 on dist-test-slave-39l8"
server_key: "67fc3969840d66af4f043fd29e93fc6e"
server_key_iv: "432200e89f73354ae8862db4bbc52195"
server_key_version: "encryptionkey@0"
I20260807 00:01:19.437108 6374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:19.451886 6374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:19.456494 6374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:19.456789 6374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:19.468405 6374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34981
I20260807 00:01:19.468478 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34981 every 8 connection(s)
I20260807 00:01:19.469728 6374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:19.471083 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6374
I20260807 00:01:19.471289 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:19.471836 9009 external_mini_cluster.cc:1428] Setting key 4dd61343ae274c85652e15f8b4b9d644
I20260807 00:01:19.475001 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:19.483719 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db: Bootstrap starting.
I20260807 00:01:19.485215 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:19.474046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:56405 (local address 127.0.0.1:34981)
0807 00:01:19.474667 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:19.474747 (+ 80us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:19.474821 (+ 74us) server_negotiation.cc:407] Connection header received
0807 00:01:19.475795 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:19.475822 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:19.476300 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:19.476807 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:19.477452 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:19.478960 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:19.480344 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:19.481182 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:19.481484 (+ 302us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:19.482899 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:19.482912 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:19.482924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:19.483435 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:19.483512 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:19.483550 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:19.483772 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:19.484038 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 00:01:19.484209 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":228,"threads_started":1}
I20260807 00:01:19.487347 6462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:19.488652 6462 log.cc:824] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:19.489424 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:19.491551 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db: No bootstrap required, opened a new log
I20260807 00:01:19.497821 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:19.492272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:32821 (local address 127.0.0.1:34981)
0807 00:01:19.492424 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:19.492436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:19.492658 (+ 222us) server_negotiation.cc:407] Connection header received
0807 00:01:19.492803 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:19.492806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:19.492876 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:19.493001 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:19.493289 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:19.494279 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:19.495468 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:19.495921 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:19.496114 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:19.497152 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:19.497157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:19.497160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:19.497180 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:19.497223 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:19.497247 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:19.497394 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:19.497532 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 00:01:19.497579 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 00:01:19.497679 6462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } }
I20260807 00:01:19.498149 6462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:19.498265 6462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2d33bf94f04a9da715cbe9500594db, State: Initialized, Role: FOLLOWER
I20260807 00:01:19.498926 6462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } }
I20260807 00:01:19.499150 6462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:19.499269 6462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:19.499449 6462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:19.500663 6462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } }
I20260807 00:01:19.501112 6462 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b2d33bf94f04a9da715cbe9500594db; no voters:
I20260807 00:01:19.501535 6462 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:19.501704 6473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:19.502043 6473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [term 1 LEADER]: Becoming Leader. State: Replica: 0b2d33bf94f04a9da715cbe9500594db, State: Running, Role: LEADER
I20260807 00:01:19.502602 6473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } }
I20260807 00:01:19.503306 6462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:19.504890 6475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b2d33bf94f04a9da715cbe9500594db. Latest consensus state: current_term: 1 leader_uuid: "0b2d33bf94f04a9da715cbe9500594db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } } }
I20260807 00:01:19.505106 6475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:19.505784 6487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:19.505726 6476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b2d33bf94f04a9da715cbe9500594db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2d33bf94f04a9da715cbe9500594db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34981 } } }
I20260807 00:01:19.505954 6476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:19.509805 6487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:19.518152 6487 catalog_manager.cc:1347] Generated new cluster ID: 413f9f41d6c64eadb69d10822be4c3b7
I20260807 00:01:19.518270 6487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:19.530284 6487 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:19.531695 6487 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:19.539170 6487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b2d33bf94f04a9da715cbe9500594db: Generated new TSK 0
I20260807 00:01:19.540153 6487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:19.550019 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6374
W20260807 00:01:19.572144 6349 connection.cc:535] client connection to 127.0.0.1:34981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34981): Connection reset by peer (error 104)
2026-08-07T00:01:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:19 dist-test-slave-39l8 krb5kdc[6508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:19 dist-test-slave-39l8 krb5kdc[6508](info): set up 2 sockets
Aug 07 00:01:19 dist-test-slave-39l8 krb5kdc[6508](info): commencing operation
krb5kdc: starting...
W20260807 00:01:20.936939 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:21 dist-test-slave-39l8 krb5kdc[6508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 00:01:21.059919 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:21.171386 6531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:21.171695 6531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:21.171775 6531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:21.175836 6531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:21.175935 6531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:21.175974 6531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:21.176012 6531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:21.176043 6531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:21.181332 6531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:21.182417 6531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:21.183490 6531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:21.191958 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:21.191954 6539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:21.192008 6538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:21.192220 6531 server_base.cc:1034] running on GCE node
I20260807 00:01:21.192907 6531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:21.193960 6531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:21.195143 6531 hybrid_clock.cc:648] HybridClock initialized: now 1786060881195120 us; error 45 us; skew 500 ppm
Aug 07 00:01:21 dist-test-slave-39l8 krb5kdc[6508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:21.197870 6531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 00:01:21.198762 6531 webserver.cc:460] Webserver started at http://127.0.0.1:35629/ using document root <none> and password file <none>
I20260807 00:01:21.199316 6531 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:21.199406 6531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:21.199635 6531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:21.201620 6531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "f7cf93f4c6f6410197509615d15914f0"
format_stamp: "Formatted at 2026-08-07 00:01:21 on dist-test-slave-39l8"
server_key: "d5fceacc2f860e1bd81cb2e4b6704733"
server_key_iv: "79b1031bd71c7b5bc388ac39b8762fa0"
server_key_version: "encryptionkey@0"
I20260807 00:01:21.202193 6531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "f7cf93f4c6f6410197509615d15914f0"
format_stamp: "Formatted at 2026-08-07 00:01:21 on dist-test-slave-39l8"
server_key: "d5fceacc2f860e1bd81cb2e4b6704733"
server_key_iv: "79b1031bd71c7b5bc388ac39b8762fa0"
server_key_version: "encryptionkey@0"
I20260807 00:01:21.205802 6531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 00:01:21.208148 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:21.209312 6531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:01:21.209451 6531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "f7cf93f4c6f6410197509615d15914f0"
format_stamp: "Formatted at 2026-08-07 00:01:21 on dist-test-slave-39l8"
server_key: "d5fceacc2f860e1bd81cb2e4b6704733"
server_key_iv: "79b1031bd71c7b5bc388ac39b8762fa0"
server_key_version: "encryptionkey@0"
I20260807 00:01:21.209555 6531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:21.213990 6531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:21.216770 6531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:21.216955 6531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:21.223927 6531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40757
I20260807 00:01:21.223950 6610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40757 every 8 connection(s)
I20260807 00:01:21.224779 6531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:21.226118 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6531
I20260807 00:01:21.226305 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:21.226578 9009 external_mini_cluster.cc:1428] Setting key ffd6c0e605ac2431f23698ce9c5a6d19
I20260807 00:01:21.227797 6611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:21.231670 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:21.227937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:50829 (local address 127.0.0.1:40757)
0807 00:01:21.228327 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:21.228345 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:21.228400 (+ 55us) server_negotiation.cc:407] Connection header received
0807 00:01:21.229010 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:21.229025 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:21.229300 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:21.229564 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:21.230399 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:21.230403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:21.230414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:21.230424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:21.230472 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:21.230490 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:21.230593 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:21.230824 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 00:01:21.231013 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260807 00:01:21.234294 6611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0: Bootstrap starting.
I20260807 00:01:21.234671 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:21.237123 6611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:21.238112 6611 log.cc:824] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:21.238838 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:21.237003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:56899 (local address 127.0.0.1:40757)
0807 00:01:21.237144 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:21.237150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:21.237407 (+ 257us) server_negotiation.cc:407] Connection header received
0807 00:01:21.237527 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:21.237530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:21.237560 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:21.237623 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:21.238336 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:21.238339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:21.238341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:21.238353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:21.238377 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:21.238387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:21.238456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:21.238606 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 00:01:21.238648 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 00:01:21.240168 6611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0: No bootstrap required, opened a new log
I20260807 00:01:21.243620 6611 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260807 00:01:21.243827 6611 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:21.243862 6611 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cf93f4c6f6410197509615d15914f0, State: Initialized, Role: FOLLOWER
I20260807 00:01:21.244283 6611 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260807 00:01:21.244467 6611 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:21.244557 6611 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:21.244685 6611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:21.245559 6611 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260807 00:01:21.245894 6611 leader_election.cc:302] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cf93f4c6f6410197509615d15914f0; no voters:
I20260807 00:01:21.246217 6611 leader_election.cc:288] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:21.246302 6623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:21.246580 6623 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [term 1 LEADER]: Becoming Leader. State: Replica: f7cf93f4c6f6410197509615d15914f0, State: Running, Role: LEADER
I20260807 00:01:21.246968 6623 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260807 00:01:21.247480 6611 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:21.248776 6625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cf93f4c6f6410197509615d15914f0. Latest consensus state: current_term: 1 leader_uuid: "f7cf93f4c6f6410197509615d15914f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260807 00:01:21.248786 6624 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cf93f4c6f6410197509615d15914f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf93f4c6f6410197509615d15914f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260807 00:01:21.248906 6625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:21.248914 6624 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:21.249276 6637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:21.252499 6637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:21.258359 6637 catalog_manager.cc:1347] Generated new cluster ID: 6b12fa7a9ecf4f12abefec02b7ccc57a
I20260807 00:01:21.258428 6637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:21.286381 6637 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:21.286813 6637 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:21.293833 6637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cf93f4c6f6410197509615d15914f0: Generated new TSK 0
I20260807 00:01:21.294427 6637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:21.305172 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6531
2026-08-07T00:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1742 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:21 dist-test-slave-39l8 krb5kdc[6654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:21 dist-test-slave-39l8 krb5kdc[6654](info): set up 2 sockets
Aug 07 00:01:21 dist-test-slave-39l8 krb5kdc[6654](info): commencing operation
krb5kdc: starting...
W20260807 00:01:22.527693 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:22.652910 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:33617
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:33617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:22.766834 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:22.767107 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:22.767186 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:22.771180 6681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:22.771273 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:22.771311 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:22.771369 6681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:22.771410 6681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:22.776715 6681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:33617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:22.777822 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:22.778900 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:22.787322 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:22.787340 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:22.787340 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:22.787526 6681 server_base.cc:1034] running on GCE node
I20260807 00:01:22.788182 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:22.789168 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:22.790333 6681 hybrid_clock.cc:648] HybridClock initialized: now 1786060882790305 us; error 45 us; skew 500 ppm
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:22.798163 6681 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:22.799208 6681 webserver.cc:460] Webserver started at http://127.8.204.126:35271/ using document root <none> and password file <none>
I20260807 00:01:22.799768 6681 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:22.799860 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:22.800091 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:22.802148 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "b34c6a12a155407e969b1e42a81e5966"
format_stamp: "Formatted at 2026-08-07 00:01:22 on dist-test-slave-39l8"
server_key: "b0d9411fdf6b8a166c8d5b15df52da21"
server_key_iv: "9e0c4ae7c2c9edb5c5d1af1be2594131"
server_key_version: "encryptionkey@0"
I20260807 00:01:22.802760 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "b34c6a12a155407e969b1e42a81e5966"
format_stamp: "Formatted at 2026-08-07 00:01:22 on dist-test-slave-39l8"
server_key: "b0d9411fdf6b8a166c8d5b15df52da21"
server_key_iv: "9e0c4ae7c2c9edb5c5d1af1be2594131"
server_key_version: "encryptionkey@0"
I20260807 00:01:22.806320 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:01:22.808738 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:22.809924 6681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 00:01:22.810070 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "b34c6a12a155407e969b1e42a81e5966"
format_stamp: "Formatted at 2026-08-07 00:01:22 on dist-test-slave-39l8"
server_key: "b0d9411fdf6b8a166c8d5b15df52da21"
server_key_iv: "9e0c4ae7c2c9edb5c5d1af1be2594131"
server_key_version: "encryptionkey@0"
I20260807 00:01:22.810176 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:22.839769 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:22.843186 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:22.843400 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:22.851248 6681 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:33617
I20260807 00:01:22.851305 6766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:33617 every 8 connection(s)
I20260807 00:01:22.852258 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:22.855225 6767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:22.860201 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6681
I20260807 00:01:22.860354 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:22.860666 9009 external_mini_cluster.cc:1428] Setting key 9af36b35f541a03c46a7713ff578f00b
I20260807 00:01:22.861083 6767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: Bootstrap starting.
I20260807 00:01:22.863641 6767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:22.864612 6767 log.cc:824] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:22.866812 6767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: No bootstrap required, opened a new log
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:22.871194 6767 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } }
I20260807 00:01:22.871454 6767 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:22.871546 6767 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34c6a12a155407e969b1e42a81e5966, State: Initialized, Role: FOLLOWER
I20260807 00:01:22.872018 6767 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } }
I20260807 00:01:22.872184 6767 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:22.872274 6767 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:22.872396 6767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:22.873322 6767 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } }
I20260807 00:01:22.873668 6767 leader_election.cc:302] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34c6a12a155407e969b1e42a81e5966; no voters:
I20260807 00:01:22.874037 6767 leader_election.cc:288] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:22.874104 6773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:22.874342 6773 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [term 1 LEADER]: Becoming Leader. State: Replica: b34c6a12a155407e969b1e42a81e5966, State: Running, Role: LEADER
I20260807 00:01:22.874727 6773 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } }
I20260807 00:01:22.875378 6767 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:22.876021 6775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34c6a12a155407e969b1e42a81e5966. Latest consensus state: current_term: 1 leader_uuid: "b34c6a12a155407e969b1e42a81e5966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } } }
I20260807 00:01:22.876114 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:22.862105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36749 (local address 127.8.204.126:33617)
0807 00:01:22.862543 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:22.862558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:22.862601 (+ 43us) server_negotiation.cc:407] Connection header received
0807 00:01:22.863295 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:22.863312 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:22.863654 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:22.863977 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:22.864951 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:22.865914 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:22.867093 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:22.867540 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:22.867671 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:22.870500 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:22.870505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:22.870518 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:22.870546 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:22.872605 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:22.873119 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:22.873125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:22.873131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:22.873256 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:22.873651 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:22.873655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:22.873657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:22.874193 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:22.874443 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:22.874798 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 00:01:22.875197 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260807 00:01:22.876160 6775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:22.876528 6774 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34c6a12a155407e969b1e42a81e5966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34c6a12a155407e969b1e42a81e5966" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 33617 } } }
I20260807 00:01:22.876641 6774 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:22.879439 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260807 00:01:22.880801 6795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:22.880883 6795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:22.880944 6781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:22.881852 6781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:22.888131 6781 catalog_manager.cc:1347] Generated new cluster ID: 036f371b57724ec6aff79eae8d5b939c
I20260807 00:01:22.888214 6781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:22.890764 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:22.881441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60289 (local address 127.8.204.126:33617)
0807 00:01:22.881556 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:22.881560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:22.881922 (+ 362us) server_negotiation.cc:407] Connection header received
0807 00:01:22.882040 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:22.882043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:22.882086 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:22.882177 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:22.882872 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:22.883389 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:22.884450 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:22.884881 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:22.884999 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:22.887004 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:22.887008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:22.887010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:22.887037 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:22.888784 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:22.889322 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:22.889326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:22.889328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:22.889448 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:22.889806 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:22.889810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:22.889812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:22.890040 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:22.890194 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:22.890461 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 00:01:22.890574 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 00:01:22.902400 6781 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:22.903626 6781 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:22.922210 6781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34c6a12a155407e969b1e42a81e5966: Generated new TSK 0
I20260807 00:01:22.923034 6781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:22.931603 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6681
W20260807 00:01:22.946287 6651 connection.cc:535] client connection to 127.8.204.126:33617 recv error: Network error: failed to read from TLS socket (remote: 127.8.204.126:33617): Connection reset by peer (error 104)
2026-08-07T00:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6810](info): set up 2 sockets
Aug 07 00:01:22 dist-test-slave-39l8 krb5kdc[6810](info): commencing operation
krb5kdc: starting...
W20260807 00:01:24.125144 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:24 dist-test-slave-39l8 krb5kdc[6810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:24.245110 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40725
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:24.355242 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:24.355502 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:24.355579 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:24.359489 6826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:24.359571 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:24.359611 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:24.359668 6826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:24.359706 6826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:24.364850 6826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:40725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:24.365831 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:24.366817 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:24.375058 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:24.375062 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:24.375131 6826 server_base.cc:1034] running on GCE node
W20260807 00:01:24.375063 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:24.375869 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:24.376885 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:24.378080 6826 hybrid_clock.cc:648] HybridClock initialized: now 1786060884378067 us; error 25 us; skew 500 ppm
Aug 07 00:01:24 dist-test-slave-39l8 krb5kdc[6810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060884, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:24.380847 6826 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:24.381681 6826 webserver.cc:460] Webserver started at http://127.8.204.126:34747/ using document root <none> and password file <none>
I20260807 00:01:24.382463 6826 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:24.382565 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:24.382853 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:24.384840 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "4473b99f0256427caad8a4da217c04ae"
format_stamp: "Formatted at 2026-08-07 00:01:24 on dist-test-slave-39l8"
server_key: "be940ee2ca2f727ebe73dec6fedf15d3"
server_key_iv: "314e741d60a21d71a244586b910ddb69"
server_key_version: "encryptionkey@0"
I20260807 00:01:24.385406 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "4473b99f0256427caad8a4da217c04ae"
format_stamp: "Formatted at 2026-08-07 00:01:24 on dist-test-slave-39l8"
server_key: "be940ee2ca2f727ebe73dec6fedf15d3"
server_key_iv: "314e741d60a21d71a244586b910ddb69"
server_key_version: "encryptionkey@0"
I20260807 00:01:24.389019 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:24.391454 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:24.392642 6826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:24.392786 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "4473b99f0256427caad8a4da217c04ae"
format_stamp: "Formatted at 2026-08-07 00:01:24 on dist-test-slave-39l8"
server_key: "be940ee2ca2f727ebe73dec6fedf15d3"
server_key_iv: "314e741d60a21d71a244586b910ddb69"
server_key_version: "encryptionkey@0"
I20260807 00:01:24.392894 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:24.419255 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:24.422047 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:24.422259 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:24.429360 6826 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:40725
I20260807 00:01:24.429383 6912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:40725 every 8 connection(s)
I20260807 00:01:24.430276 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:24.432108 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6826
I20260807 00:01:24.432251 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:24.432605 9009 external_mini_cluster.cc:1428] Setting key 94be24c8e00558549459f4ecd4f53ff9
I20260807 00:01:24.433460 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:24.439770 6913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae: Bootstrap starting.
I20260807 00:01:24.441344 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:24.433933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40949 (local address 127.8.204.126:40725)
0807 00:01:24.434358 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:24.434373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:24.434408 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:24.435131 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:24.435149 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:24.435468 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:24.435734 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:24.436069 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:24.437021 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:24.438231 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:24.438674 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:24.438865 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:24.439699 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:24.439704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:24.439714 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:24.440107 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:24.440172 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:24.440197 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:24.440296 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:24.440531 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 00:01:24.440664 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
I20260807 00:01:24.442181 6913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:24.442981 6913 log.cc:824] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:24.444303 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:24.444988 6913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae: No bootstrap required, opened a new log
I20260807 00:01:24.449529 6913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } }
I20260807 00:01:24.449820 6913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:24.449908 6913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4473b99f0256427caad8a4da217c04ae, State: Initialized, Role: FOLLOWER
I20260807 00:01:24.450486 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:24.446271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49699 (local address 127.8.204.126:40725)
0807 00:01:24.446408 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:24.446412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:24.446529 (+ 117us) server_negotiation.cc:407] Connection header received
0807 00:01:24.446640 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:24.446642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:24.446678 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:24.446814 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:24.447120 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:24.447587 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:24.448793 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:24.449011 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:24.449087 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:24.449917 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:24.449920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:24.449924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:24.449939 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:24.449965 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:24.449979 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:24.450084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:24.450302 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 00:01:24.450338 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 00:01:24.450439 6913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } }
I20260807 00:01:24.450615 6913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:24.450696 6913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:24.450819 6913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:24.451864 6913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } }
I20260807 00:01:24.452170 6913 leader_election.cc:302] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4473b99f0256427caad8a4da217c04ae; no voters:
I20260807 00:01:24.452404 6913 leader_election.cc:288] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:24.452533 6926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:24.452770 6926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [term 1 LEADER]: Becoming Leader. State: Replica: 4473b99f0256427caad8a4da217c04ae, State: Running, Role: LEADER
I20260807 00:01:24.453104 6926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } }
I20260807 00:01:24.453565 6913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:24.454627 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4473b99f0256427caad8a4da217c04ae. Latest consensus state: current_term: 1 leader_uuid: "4473b99f0256427caad8a4da217c04ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } } }
I20260807 00:01:24.454798 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:24.454955 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4473b99f0256427caad8a4da217c04ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4473b99f0256427caad8a4da217c04ae" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 40725 } } }
I20260807 00:01:24.455114 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:24.455451 6941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:24.458213 6941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:24.463764 6941 catalog_manager.cc:1347] Generated new cluster ID: 13d34a641d934dbeb6f37dc2bf2ebc25
I20260807 00:01:24.463827 6941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:24.472867 6941 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:24.473675 6941 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:24.480280 6941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4473b99f0256427caad8a4da217c04ae: Generated new TSK 0
I20260807 00:01:24.480870 6941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:24.490106 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6826
W20260807 00:01:24.506963 6807 connection.cc:535] client connection to 127.8.204.126:40725 recv error: Network error: failed to read from TLS socket (remote: 127.8.204.126:40725): Connection reset by peer (error 104)
2026-08-07T00:01:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:24 dist-test-slave-39l8 krb5kdc[6956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:24 dist-test-slave-39l8 krb5kdc[6956](info): set up 2 sockets
Aug 07 00:01:24 dist-test-slave-39l8 krb5kdc[6956](info): commencing operation
krb5kdc: starting...
W20260807 00:01:25.693192 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:25 dist-test-slave-39l8 krb5kdc[6956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:25.814668 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41457
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:41457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:25.925596 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:25.925855 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:25.925897 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:25.929876 6975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:25.929934 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:25.929955 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:25.929971 6975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:25.929986 6975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:25.935212 6975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:41457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:25.936236 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:25.937254 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:25.945518 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:25.945598 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:25.945602 6989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:25.945837 6975 server_base.cc:1034] running on GCE node
I20260807 00:01:25.946444 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:25.947571 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:25.948762 6975 hybrid_clock.cc:648] HybridClock initialized: now 1786060885948739 us; error 45 us; skew 500 ppm
Aug 07 00:01:25 dist-test-slave-39l8 krb5kdc[6956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060885, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:25.951454 6975 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:25.952365 6975 webserver.cc:460] Webserver started at http://127.8.204.126:44479/ using document root <none> and password file <none>
I20260807 00:01:25.952922 6975 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:25.953013 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:25.953256 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:25.955382 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4"
format_stamp: "Formatted at 2026-08-07 00:01:25 on dist-test-slave-39l8"
server_key: "990845ae7c7bd1c63e72de68fdc88606"
server_key_iv: "4fbdb0c9555729549771e964ad28c324"
server_key_version: "encryptionkey@0"
I20260807 00:01:25.955952 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4"
format_stamp: "Formatted at 2026-08-07 00:01:25 on dist-test-slave-39l8"
server_key: "990845ae7c7bd1c63e72de68fdc88606"
server_key_iv: "4fbdb0c9555729549771e964ad28c324"
server_key_version: "encryptionkey@0"
I20260807 00:01:25.959470 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 00:01:25.961758 7002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:25.963004 6975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 00:01:25.963145 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4"
format_stamp: "Formatted at 2026-08-07 00:01:25 on dist-test-slave-39l8"
server_key: "990845ae7c7bd1c63e72de68fdc88606"
server_key_iv: "4fbdb0c9555729549771e964ad28c324"
server_key_version: "encryptionkey@0"
I20260807 00:01:25.963253 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:25.967746 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:25.970420 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:25.970633 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:25.977674 6975 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:41457
I20260807 00:01:25.977684 7070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:41457 every 8 connection(s)
I20260807 00:01:25.978572 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:25.980999 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 6975
I20260807 00:01:25.981151 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:25.981482 9009 external_mini_cluster.cc:1428] Setting key b3226f845651fbec1458f442d7e2ac2c
I20260807 00:01:25.981559 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:25.986963 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:25.982901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59279 (local address 127.8.204.126:41457)
0807 00:01:25.983353 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:25.983368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:25.983404 (+ 36us) server_negotiation.cc:407] Connection header received
0807 00:01:25.984319 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:25.984337 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:25.984607 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:25.984929 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:25.985737 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:25.985741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:25.985752 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:25.985763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:25.985816 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:25.985847 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:25.986017 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:25.986096 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 00:01:25.986252 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260807 00:01:25.988296 7071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4: Bootstrap starting.
I20260807 00:01:25.989890 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:25.990842 7071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:25.991632 7071 log.cc:824] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:25.993548 7071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4: No bootstrap required, opened a new log
I20260807 00:01:25.993876 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:25.992227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44451 (local address 127.8.204.126:41457)
0807 00:01:25.992335 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:25.992339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:25.992518 (+ 179us) server_negotiation.cc:407] Connection header received
0807 00:01:25.992666 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:25.992669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:25.992713 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:25.992798 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:25.993487 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:25.993490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:25.993493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:25.993505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:25.993530 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:25.993556 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:25.993655 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:25.993729 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 00:01:25.993766 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 00:01:25.997262 7071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } }
I20260807 00:01:25.997469 7071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:25.997551 7071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bcf39e214b4ca0a55cf2fbdd63dba4, State: Initialized, Role: FOLLOWER
I20260807 00:01:25.998039 7071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } }
I20260807 00:01:25.998165 7071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:25.998252 7071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:25.998365 7071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:25.999318 7071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } }
I20260807 00:01:25.999650 7071 leader_election.cc:302] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45bcf39e214b4ca0a55cf2fbdd63dba4; no voters:
I20260807 00:01:25.999953 7071 leader_election.cc:288] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:26.000079 7085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:26.000312 7085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [term 1 LEADER]: Becoming Leader. State: Replica: 45bcf39e214b4ca0a55cf2fbdd63dba4, State: Running, Role: LEADER
I20260807 00:01:26.000665 7085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } }
I20260807 00:01:26.001142 7071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:26.002305 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45bcf39e214b4ca0a55cf2fbdd63dba4. Latest consensus state: current_term: 1 leader_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } } }
I20260807 00:01:26.002458 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:26.002863 7095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:26.002876 7087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bcf39e214b4ca0a55cf2fbdd63dba4" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41457 } } }
I20260807 00:01:26.003013 7087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:26.006494 7095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:26.012326 7095 catalog_manager.cc:1347] Generated new cluster ID: 8f506d6d5ee84449ad5963e8018c4428
I20260807 00:01:26.012418 7095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:26.023698 7095 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:26.023983 7095 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:26.031301 7095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45bcf39e214b4ca0a55cf2fbdd63dba4: Generated new TSK 0
I20260807 00:01:26.031914 7095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:26.040395 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 6975
2026-08-07T00:01:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1544 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:26 dist-test-slave-39l8 krb5kdc[7114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:26 dist-test-slave-39l8 krb5kdc[7114](info): set up 2 sockets
Aug 07 00:01:26 dist-test-slave-39l8 krb5kdc[7114](info): commencing operation
krb5kdc: starting...
W20260807 00:01:27.247797 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:27.377619 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:46833
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:27.491073 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:27.491364 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:27.491437 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:27.496057 7132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:27.496165 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:27.496210 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:27.496274 7132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:27.496321 7132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:27.501881 7132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:27.503000 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:27.504019 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:27.512660 7138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:27.512678 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:27.512660 7139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:27.513033 7132 server_base.cc:1034] running on GCE node
I20260807 00:01:27.513588 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:27.514632 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:27.515843 7132 hybrid_clock.cc:648] HybridClock initialized: now 1786060887515818 us; error 40 us; skew 500 ppm
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:27.518586 7132 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:27.519419 7132 webserver.cc:460] Webserver started at http://127.8.204.126:45959/ using document root <none> and password file <none>
I20260807 00:01:27.519922 7132 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:27.519980 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:27.520135 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:27.522018 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "bf5089a7b059491fa22fd4b5fc59c79f"
format_stamp: "Formatted at 2026-08-07 00:01:27 on dist-test-slave-39l8"
server_key: "6cb442c536ea07d63331710dd7db52ab"
server_key_iv: "d338e464dd2507857151d001e2285880"
server_key_version: "encryptionkey@0"
I20260807 00:01:27.522511 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "bf5089a7b059491fa22fd4b5fc59c79f"
format_stamp: "Formatted at 2026-08-07 00:01:27 on dist-test-slave-39l8"
server_key: "6cb442c536ea07d63331710dd7db52ab"
server_key_iv: "d338e464dd2507857151d001e2285880"
server_key_version: "encryptionkey@0"
I20260807 00:01:27.525992 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260807 00:01:27.528299 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:27.529458 7132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 00:01:27.529556 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "bf5089a7b059491fa22fd4b5fc59c79f"
format_stamp: "Formatted at 2026-08-07 00:01:27 on dist-test-slave-39l8"
server_key: "6cb442c536ea07d63331710dd7db52ab"
server_key_iv: "d338e464dd2507857151d001e2285880"
server_key_version: "encryptionkey@0"
I20260807 00:01:27.529635 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:27.547273 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:27.550434 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:27.550582 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:27.557719 7132 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:46833
I20260807 00:01:27.557736 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:46833 every 8 connection(s)
I20260807 00:01:27.558645 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:27.561327 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:27.564323 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7132
I20260807 00:01:27.564473 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:27.564831 9009 external_mini_cluster.cc:1428] Setting key 469e68ef1cc02dfc191b5b27fdf17881
I20260807 00:01:27.567427 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: Bootstrap starting.
I20260807 00:01:27.569577 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:27.570298 7220 log.cc:824] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:27.572173 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: No bootstrap required, opened a new log
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:27.575248 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } }
I20260807 00:01:27.575486 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:27.575565 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5089a7b059491fa22fd4b5fc59c79f, State: Initialized, Role: FOLLOWER
I20260807 00:01:27.576136 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } }
I20260807 00:01:27.576299 7220 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:27.576367 7220 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:27.576467 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:27.577559 7220 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } }
I20260807 00:01:27.577966 7220 leader_election.cc:302] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5089a7b059491fa22fd4b5fc59c79f; no voters:
I20260807 00:01:27.578293 7220 leader_election.cc:288] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:27.578401 7227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:27.578686 7227 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [term 1 LEADER]: Becoming Leader. State: Replica: bf5089a7b059491fa22fd4b5fc59c79f, State: Running, Role: LEADER
I20260807 00:01:27.579250 7227 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } }
I20260807 00:01:27.579710 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:27.579967 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:27.566229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33171 (local address 127.8.204.126:46833)
0807 00:01:27.566656 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:27.566670 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:27.566751 (+ 81us) server_negotiation.cc:407] Connection header received
0807 00:01:27.567399 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:27.567416 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:27.567792 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:27.568113 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:27.568951 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:27.569973 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:27.570958 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:27.571421 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:27.571596 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:27.574113 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:27.574118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:27.574131 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:27.574158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:27.576769 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:27.577182 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:27.577190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:27.577198 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:27.577352 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:27.577622 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:27.577625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:27.577627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:27.578157 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:27.578359 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:27.578754 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 00:01:27.579054 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260807 00:01:27.580456 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } } }
I20260807 00:01:27.580598 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:27.580816 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5089a7b059491fa22fd4b5fc59c79f. Latest consensus state: current_term: 1 leader_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5089a7b059491fa22fd4b5fc59c79f" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 46833 } } }
I20260807 00:01:27.580914 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:27.583199 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260807 00:01:27.585407 7253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:27.585492 7253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:27.585554 7243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:27.586445 7243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:27.592456 7243 catalog_manager.cc:1347] Generated new cluster ID: 44b9927a47644265a0abc987a0964b5e
I20260807 00:01:27.592530 7243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:27.595293 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:27.585967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41351 (local address 127.8.204.126:46833)
0807 00:01:27.586086 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:27.586090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:27.586370 (+ 280us) server_negotiation.cc:407] Connection header received
0807 00:01:27.586525 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:27.586530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:27.586584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:27.586677 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:27.587329 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:27.587992 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:27.589103 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:27.589493 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:27.589623 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:27.591961 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:27.591964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:27.591967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:27.592003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:27.593604 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:27.594047 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:27.594052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:27.594058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:27.594129 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:27.594407 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:27.594412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:27.594413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:27.594586 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:27.594747 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:27.594986 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 00:01:27.595113 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:01:27.610427 7243 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:27.611373 7243 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:27.616614 7243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5089a7b059491fa22fd4b5fc59c79f: Generated new TSK 0
I20260807 00:01:27.617503 7243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:27.629935 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7132
2026-08-07T00:01:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1591 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7270](info): set up 2 sockets
Aug 07 00:01:27 dist-test-slave-39l8 krb5kdc[7270](info): commencing operation
krb5kdc: starting...
W20260807 00:01:28.888108 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:28 dist-test-slave-39l8 krb5kdc[7270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:29.011381 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41469
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:41469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:29.122192 7285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:29.122498 7285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:29.122591 7285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:29.126540 7285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:29.126636 7285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:29.126674 7285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:29.126750 7285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:29.126806 7285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:29.132020 7285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:41469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:29.133026 7285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:29.134063 7285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:29.142426 7296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:29.142418 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:29.142428 7299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:29.142596 7285 server_base.cc:1034] running on GCE node
I20260807 00:01:29.143340 7285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:29.144366 7285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:29.145557 7285 hybrid_clock.cc:648] HybridClock initialized: now 1786060889145544 us; error 24 us; skew 500 ppm
Aug 07 00:01:29 dist-test-slave-39l8 krb5kdc[7270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:29.148293 7285 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:29.149111 7285 webserver.cc:460] Webserver started at http://127.8.204.126:42713/ using document root <none> and password file <none>
I20260807 00:01:29.149652 7285 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:29.149755 7285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:29.149981 7285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:29.152005 7285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "330d33e207384e45b109e73886700673"
format_stamp: "Formatted at 2026-08-07 00:01:29 on dist-test-slave-39l8"
server_key: "87122918e506e7bdbfa5d07d31024fa2"
server_key_iv: "3c2ced5aff87a411c3ce2519cc9dbfb5"
server_key_version: "encryptionkey@0"
I20260807 00:01:29.152573 7285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "330d33e207384e45b109e73886700673"
format_stamp: "Formatted at 2026-08-07 00:01:29 on dist-test-slave-39l8"
server_key: "87122918e506e7bdbfa5d07d31024fa2"
server_key_iv: "3c2ced5aff87a411c3ce2519cc9dbfb5"
server_key_version: "encryptionkey@0"
I20260807 00:01:29.156071 7285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 00:01:29.158434 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:29.159670 7285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:29.159824 7285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "330d33e207384e45b109e73886700673"
format_stamp: "Formatted at 2026-08-07 00:01:29 on dist-test-slave-39l8"
server_key: "87122918e506e7bdbfa5d07d31024fa2"
server_key_iv: "3c2ced5aff87a411c3ce2519cc9dbfb5"
server_key_version: "encryptionkey@0"
I20260807 00:01:29.159938 7285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:29.183373 7285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:29.186081 7285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:29.186290 7285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:29.193204 7285 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:41469
I20260807 00:01:29.193285 7368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:41469 every 8 connection(s)
I20260807 00:01:29.194159 7285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:29.196950 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:29.199605 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7285
I20260807 00:01:29.199754 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:29.200066 9009 external_mini_cluster.cc:1428] Setting key ad380332cf2ccd97958ffa571b286588
I20260807 00:01:29.203610 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673: Bootstrap starting.
I20260807 00:01:29.206000 7371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:29.206890 7371 log.cc:824] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:29.208894 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:29.201463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50441 (local address 127.8.204.126:41469)
0807 00:01:29.202038 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:29.202052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:29.202085 (+ 33us) server_negotiation.cc:407] Connection header received
0807 00:01:29.202769 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:29.202799 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:29.203115 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:29.203439 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:29.203874 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:29.204817 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:29.205738 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:29.206247 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:29.206428 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:29.207304 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:29.207316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:29.207330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:29.207687 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:29.207742 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:29.207764 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:29.207879 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:29.208030 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 00:01:29.208237 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260807 00:01:29.209039 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673: No bootstrap required, opened a new log
I20260807 00:01:29.211666 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:29.213279 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } }
I20260807 00:01:29.213549 7371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:29.213665 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330d33e207384e45b109e73886700673, State: Initialized, Role: FOLLOWER
I20260807 00:01:29.214285 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } }
I20260807 00:01:29.214478 7371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:29.214597 7371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:29.214764 7371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:29.215950 7371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } }
I20260807 00:01:29.216364 7371 leader_election.cc:302] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330d33e207384e45b109e73886700673; no voters:
I20260807 00:01:29.216760 7371 leader_election.cc:288] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:29.216955 7385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:29.217293 7385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [term 1 LEADER]: Becoming Leader. State: Replica: 330d33e207384e45b109e73886700673, State: Running, Role: LEADER
I20260807 00:01:29.217640 7385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } }
I20260807 00:01:29.218393 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:29.218420 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:29.213616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33007 (local address 127.8.204.126:41469)
0807 00:01:29.213748 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:29.213752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:29.213943 (+ 191us) server_negotiation.cc:407] Connection header received
0807 00:01:29.214089 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:29.214092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:29.214128 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:29.214202 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:29.214574 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:29.215181 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:29.216382 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:29.216652 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:29.216886 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:29.217794 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:29.217797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:29.217800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:29.217813 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:29.217840 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:29.217852 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:29.217924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:29.218236 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 00:01:29.218284 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 00:01:29.219385 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 330d33e207384e45b109e73886700673. Latest consensus state: current_term: 1 leader_uuid: "330d33e207384e45b109e73886700673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } } }
I20260807 00:01:29.219511 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:29.219756 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "330d33e207384e45b109e73886700673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d33e207384e45b109e73886700673" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 41469 } } }
I20260807 00:01:29.219895 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:29.220033 7399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:29.223209 7399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:29.229255 7399 catalog_manager.cc:1347] Generated new cluster ID: 58273ecd12d74c6bbee1c68d8fe6a1bc
I20260807 00:01:29.229331 7399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:29.245342 7399 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:29.246261 7399 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:29.256235 7399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 330d33e207384e45b109e73886700673: Generated new TSK 0
I20260807 00:01:29.256937 7399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:29.269906 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7285
2026-08-07T00:01:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:29 dist-test-slave-39l8 krb5kdc[7417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:29 dist-test-slave-39l8 krb5kdc[7417](info): set up 2 sockets
Aug 07 00:01:29 dist-test-slave-39l8 krb5kdc[7417](info): commencing operation
krb5kdc: starting...
W20260807 00:01:30.465490 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:30 dist-test-slave-39l8 krb5kdc[7417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:30.583468 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35243
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:30.696703 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:30.696944 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:30.696985 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:30.701018 7433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:30.701113 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:30.701134 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:30.701207 7433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:30.701234 7433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:30.706341 7433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:30.707362 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:30.708323 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:30.716816 7442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:30.716837 7440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:30.716837 7439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:30.717090 7433 server_base.cc:1034] running on GCE node
I20260807 00:01:30.717698 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:30.718679 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:30.719908 7433 hybrid_clock.cc:648] HybridClock initialized: now 1786060890719891 us; error 30 us; skew 500 ppm
Aug 07 00:01:30 dist-test-slave-39l8 krb5kdc[7417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060890, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:30.722894 7433 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:30.723721 7433 webserver.cc:460] Webserver started at http://127.8.204.126:45167/ using document root <none> and password file <none>
I20260807 00:01:30.724270 7433 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:30.724332 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:30.724545 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:30.726529 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "9f84a3999c914f48aec1e51d2e3789d1"
format_stamp: "Formatted at 2026-08-07 00:01:30 on dist-test-slave-39l8"
server_key: "dadcbddbc5e69218a964ba42cbe50511"
server_key_iv: "d129e586b2fecc0621ccd660fa3cc8fc"
server_key_version: "encryptionkey@0"
I20260807 00:01:30.727139 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "9f84a3999c914f48aec1e51d2e3789d1"
format_stamp: "Formatted at 2026-08-07 00:01:30 on dist-test-slave-39l8"
server_key: "dadcbddbc5e69218a964ba42cbe50511"
server_key_iv: "d129e586b2fecc0621ccd660fa3cc8fc"
server_key_version: "encryptionkey@0"
I20260807 00:01:30.730588 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 00:01:30.732911 7455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:30.734087 7433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:30.734230 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "9f84a3999c914f48aec1e51d2e3789d1"
format_stamp: "Formatted at 2026-08-07 00:01:30 on dist-test-slave-39l8"
server_key: "dadcbddbc5e69218a964ba42cbe50511"
server_key_iv: "d129e586b2fecc0621ccd660fa3cc8fc"
server_key_version: "encryptionkey@0"
I20260807 00:01:30.734335 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:30.738873 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:30.741621 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:30.741809 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:30.748659 7433 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:35243
I20260807 00:01:30.748687 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:35243 every 8 connection(s)
I20260807 00:01:30.749536 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:30.749984 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7433
I20260807 00:01:30.750128 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:30.750418 9009 external_mini_cluster.cc:1428] Setting key f0f697f1efccb832834e9068e1cf2f3b
I20260807 00:01:30.752657 7529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:30.755581 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:30.751853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38495 (local address 127.8.204.126:35243)
0807 00:01:30.752270 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:30.752294 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:30.752328 (+ 34us) server_negotiation.cc:407] Connection header received
0807 00:01:30.752999 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:30.753016 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:30.753283 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:30.753546 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:30.754389 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:30.754394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:30.754406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:30.754416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:30.754465 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:30.754483 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:30.754588 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:30.754775 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 00:01:30.754946 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":161,"threads_started":1}
I20260807 00:01:30.758363 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:30.759219 7529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1: Bootstrap starting.
I20260807 00:01:30.761807 7529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:30.762214 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:30.760590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52621 (local address 127.8.204.126:35243)
0807 00:01:30.760701 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:30.760707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:30.760854 (+ 147us) server_negotiation.cc:407] Connection header received
0807 00:01:30.760962 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:30.760964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:30.760991 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:30.761053 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:30.761736 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:30.761738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:30.761740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 00:01:30.761750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:30.761772 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 00:01:30.761782 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:30.761848 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:30.762017 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 00:01:30.762068 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 00:01:30.762679 7529 log.cc:824] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:30.764528 7529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1: No bootstrap required, opened a new log
I20260807 00:01:30.768065 7529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } }
I20260807 00:01:30.768311 7529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 00:01:30.768395 7529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f84a3999c914f48aec1e51d2e3789d1, State: Initialized, Role: FOLLOWER
I20260807 00:01:30.768819 7529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } }
I20260807 00:01:30.768972 7529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 00:01:30.769060 7529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 00:01:30.769172 7529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:30.770035 7529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } }
I20260807 00:01:30.770382 7529 leader_election.cc:302] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f84a3999c914f48aec1e51d2e3789d1; no voters:
I20260807 00:01:30.770692 7529 leader_election.cc:288] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 00:01:30.770798 7540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:30.771051 7540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9f84a3999c914f48aec1e51d2e3789d1, State: Running, Role: LEADER
I20260807 00:01:30.771350 7540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } }
I20260807 00:01:30.772192 7529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:30.772615 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f84a3999c914f48aec1e51d2e3789d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } } }
I20260807 00:01:30.772744 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:30.773363 7541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f84a3999c914f48aec1e51d2e3789d1. Latest consensus state: current_term: 1 leader_uuid: "9f84a3999c914f48aec1e51d2e3789d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f84a3999c914f48aec1e51d2e3789d1" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 35243 } } }
I20260807 00:01:30.773486 7541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:30.773801 7545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:30.777045 7545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:30.782581 7545 catalog_manager.cc:1347] Generated new cluster ID: 1a64bc9b4f2146f3a711908c9c7303f0
I20260807 00:01:30.782661 7545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:30.794589 7545 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:30.794899 7545 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:30.802959 7545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f84a3999c914f48aec1e51d2e3789d1: Generated new TSK 0
I20260807 00:01:30.803558 7545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 00:01:30.808871 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7433
2026-08-07T00:01:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1537 ms)
[----------] 18 tests from AuthTokenIssuingTest (29031 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:30 dist-test-slave-39l8 krb5kdc[7568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:30 dist-test-slave-39l8 krb5kdc[7568](info): set up 2 sockets
Aug 07 00:01:30 dist-test-slave-39l8 krb5kdc[7568](info): commencing operation
krb5kdc: starting...
W20260807 00:01:32.010289 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:32.134825 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36473
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:32.249166 7583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:32.249452 7583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:32.249476 7583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:32.249514 7583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:32.253588 7583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:32.253654 7583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:32.253674 7583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:32.253690 7583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:32.253705 7583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:32.259078 7583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:36473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:32.260282 7583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:32.261334 7583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:32.269991 7590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:32.269987 7593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:32.270036 7583 server_base.cc:1034] running on GCE node
W20260807 00:01:32.269987 7591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:32.270915 7583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:32.272045 7583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:32.273248 7583 hybrid_clock.cc:648] HybridClock initialized: now 1786060892273222 us; error 40 us; skew 500 ppm
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:32.275995 7583 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:32.276907 7583 webserver.cc:460] Webserver started at http://127.8.204.126:36809/ using document root <none> and password file <none>
I20260807 00:01:32.277477 7583 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:32.277572 7583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:32.277808 7583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:32.279893 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "16e4015198904560b8dcbe6314761547"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "28bfeb969b139235c3f5285979e3dabe"
server_key_iv: "68f4faef4f0749ccf41d393bb44c9f4f"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.280608 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "16e4015198904560b8dcbe6314761547"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "28bfeb969b139235c3f5285979e3dabe"
server_key_iv: "68f4faef4f0749ccf41d393bb44c9f4f"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.284349 7583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:32.286684 7608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:32.287940 7583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:32.288098 7583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "16e4015198904560b8dcbe6314761547"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "28bfeb969b139235c3f5285979e3dabe"
server_key_iv: "68f4faef4f0749ccf41d393bb44c9f4f"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.288218 7583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:32.303782 7583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:32.306459 7583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:32.306669 7583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:32.313887 7583 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:36473
I20260807 00:01:32.314169 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:36473 every 8 connection(s)
I20260807 00:01:32.314900 7583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:32.317646 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:32.321293 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7583
I20260807 00:01:32.321437 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:32.321382 7690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.321710 9009 external_mini_cluster.cc:1428] Setting key 0295c1bcb139b81fe9df027353c9f094
WARNING: no policy specified for kudu/127.8.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.125@KRBTEST.COM" created.
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:32.340528 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.324921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.8.204.126:36473)
0807 00:01:32.325272 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.325287 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.325334 (+ 47us) server_negotiation.cc:407] Connection header received
0807 00:01:32.325816 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.325824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.326062 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.326284 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.327140 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.328671 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.329651 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.330301 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.330473 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.335510 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.335517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.335537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.335559 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.337262 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.337771 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.337777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.337779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.337889 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.338228 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.338232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.338233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.338624 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.338800 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.339188 (+ 388us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.339447 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260807 00:01:32.340528 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.323962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36473 (local address 127.0.0.1:60486)
0807 00:01:32.324489 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.324544 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.325500 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.326280 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.326285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.326632 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.326966 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.326980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.328806 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.328809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.329527 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.329536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.330458 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.330461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.330585 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.331231 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.331247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.335332 (+ 4085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.337399 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.337408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.337418 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.337654 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.338003 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.338011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.338013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.338126 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.338772 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.338779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.338928 (+ 149us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.339113 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.339499 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":113,"thread_start_us":140,"threads_started":1}
I20260807 00:01:32.343724 7690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.344483 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.344211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50924)
0807 00:01:32.344350 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.344420 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 00:01:32.344679 7612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
I20260807 00:01:32.345664 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.345366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50926)
0807 00:01:32.345518 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.345590 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 00:01:32.345824 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:43633: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
I20260807 00:01:32.349193 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:43633
--webserver_interface=127.8.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:01:32.395076 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } attempt: 1
I20260807 00:01:32.395962 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.395648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50932)
0807 00:01:32.395816 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.395882 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 00:01:32.396840 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.396629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50936)
0807 00:01:32.396735 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.396783 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 00:01:32.397002 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:43633: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
W20260807 00:01:32.464692 7701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:32.464982 7701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:32.465013 7701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:32.465049 7701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 00:01:32.465214 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } attempt: 2
I20260807 00:01:32.466065 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.465753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50946)
0807 00:01:32.465919 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.465988 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 00:01:32.466892 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.466651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50958)
0807 00:01:32.466784 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.466841 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 00:01:32.467082 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:43633: Network error: Client connection negotiation failed: client connection to 127.8.204.125:43633: connect: Connection refused (error 111)
W20260807 00:01:32.469668 7701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:32.469725 7701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:32.469745 7701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:32.469789 7701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:32.469822 7701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:32.475245 7701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:32.476290 7701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:32.477514 7701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:32.486372 7711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:32.486372 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:32.486460 7701 server_base.cc:1034] running on GCE node
W20260807 00:01:32.486397 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:32.487177 7701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:32.488296 7701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:32.489509 7701 hybrid_clock.cc:648] HybridClock initialized: now 1786060892489486 us; error 42 us; skew 500 ppm
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:32.492537 7701 init.cc:377] Logged in from keytab as kudu/127.8.204.125@KRBTEST.COM (short username kudu)
I20260807 00:01:32.493378 7701 webserver.cc:460] Webserver started at http://127.8.204.125:33849/ using document root <none> and password file <none>
I20260807 00:01:32.493902 7701 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:32.493960 7701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:32.494134 7701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:32.496209 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data/instance:
uuid: "7a8896bbbad44ecc9125515aa8d13b1c"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "7afe99a6621b080d690917f199606151"
server_key_iv: "a239d46b667dc2d83931e5185eb5cb5d"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.496721 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal/instance:
uuid: "7a8896bbbad44ecc9125515aa8d13b1c"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "7afe99a6621b080d690917f199606151"
server_key_iv: "a239d46b667dc2d83931e5185eb5cb5d"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.500432 7701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260807 00:01:32.502875 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:32.504098 7701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 00:01:32.504242 7701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "7a8896bbbad44ecc9125515aa8d13b1c"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "7afe99a6621b080d690917f199606151"
server_key_iv: "a239d46b667dc2d83931e5185eb5cb5d"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.504349 7701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:32.525487 7701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:32.528981 7701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:32.529332 7701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:32.536820 7701 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.125:43633
I20260807 00:01:32.537122 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.125:43633 every 8 connection(s)
I20260807 00:01:32.537909 7701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
I20260807 00:01:32.540968 7791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:32.544819 7791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.546363 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7701
I20260807 00:01:32.546506 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-1/wal/instance
I20260807 00:01:32.546921 9009 external_mini_cluster.cc:1428] Setting key 50d4b38c4831222743233ddbb34a4b7b
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.124@KRBTEST.COM; defaulting to no policy
I20260807 00:01:32.558256 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.124@KRBTEST.COM" created.
I20260807 00:01:32.561136 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.546950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.8.204.126:36473)
0807 00:01:32.547101 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.547106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.548142 (+ 1036us) server_negotiation.cc:407] Connection header received
0807 00:01:32.549467 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.549470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.549536 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.549621 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.551025 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.551561 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.553430 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.553656 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.553739 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.556546 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.556549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.556551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.556575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.558176 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.558972 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.558977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.558980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.559076 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.559799 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.559805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.559809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.560078 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.560190 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.560793 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.560946 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 00:01:32.561887 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.547372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36473 (local address 127.0.0.1:60496)
0807 00:01:32.547937 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.548000 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.549251 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.549801 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.549808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.550150 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.550842 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.550867 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.551707 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.551714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.553299 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.553314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.553824 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.553831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.554071 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.554729 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.554754 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.556356 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.558320 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.558326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.558340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.558867 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.559209 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.559215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.559220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.559408 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.560198 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.560205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.560478 (+ 273us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.560665 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.560959 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260807 00:01:32.563766 7791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:32.569344 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.558730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50962)
0807 00:01:32.558878 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.558895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.558990 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.559591 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.559594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.559739 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.559832 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.559837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.561100 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.561104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.561979 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.561994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.562896 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.562900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.563014 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.563489 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.563501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.565086 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.567506 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.567509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.567512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.567754 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.568107 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.568110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.568112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.568211 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.568795 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.568798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.568876 (+ 78us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.568994 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.569148 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 00:01:32.569772 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.558834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.8.204.125:43633)
0807 00:01:32.559226 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.559235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.559256 (+ 21us) server_negotiation.cc:407] Connection header received
0807 00:01:32.559291 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.559294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.559432 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.559624 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.559949 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.560974 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.562153 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.562776 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.562897 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.565248 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.565261 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.565277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.565303 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.567381 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.567877 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.567884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.567885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.567987 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.568314 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.568319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.568321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.568668 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.568827 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.569102 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.569544 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"spinlock_wait_cycles":2944,"thread_start_us":130,"threads_started":1}
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:32.571615 7690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.572388 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.572120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55166)
0807 00:01:32.572246 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.572293 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 00:01:32.573279 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.572969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55176)
0807 00:01:32.573137 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.573205 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260807 00:01:32.573395 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.575260 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.565014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50970)
0807 00:01:32.565206 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.565223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.565303 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.565939 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.565943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.566131 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.566252 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.566261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.567134 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.567139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.567793 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.567806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.568881 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.568884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.568984 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.569402 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.569422 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.571833 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.573919 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.573922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.573924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.574113 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.574359 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.574361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.574363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.574433 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.574818 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.574821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.574903 (+ 82us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.574988 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.575117 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260807 00:01:32.575402 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.565139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.8.204.125:43633)
0807 00:01:32.565716 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.565726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.565741 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:32.565776 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.565779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.565824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.565929 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.566379 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.567005 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.567917 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.568204 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.568311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.572018 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.572025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.572030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.572055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.573830 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.574204 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.574207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.574208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.574280 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.574512 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.574514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.574515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.574669 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.574823 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.575047 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.575188 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
I20260807 00:01:32.575968 7791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.576663 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.576393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55188)
0807 00:01:32.576518 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.576604 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 00:01:32.576884 7723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.577015 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:34243
--webserver_interface=127.8.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:01:32.577958 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.577681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55204)
0807 00:01:32.577819 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.577881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 00:01:32.578140 7791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.604544 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 1
I20260807 00:01:32.605378 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.605091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55216)
0807 00:01:32.605241 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.605302 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 00:01:32.606134 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.605895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55228)
0807 00:01:32.606009 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.606062 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 00:01:32.606268 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.627410 7791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 1
I20260807 00:01:32.628305 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.627955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55240)
0807 00:01:32.628143 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.628211 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 00:01:32.629067 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.628830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55244)
0807 00:01:32.628949 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.629006 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 00:01:32.629206 7791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.681420 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 2
I20260807 00:01:32.682307 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.681973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55260)
0807 00:01:32.682147 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.682230 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 00:01:32.683174 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.682957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55262)
0807 00:01:32.683064 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.683121 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 00:01:32.683308 7690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
W20260807 00:01:32.690050 7803 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:32.690377 7803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:32.690439 7803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:32.690516 7803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:32.694574 7803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:32.694659 7803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:32.694698 7803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:32.694753 7803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:32.694787 7803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:32.697376 7791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 2
I20260807 00:01:32.698163 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.697835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55272)
0807 00:01:32.698009 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.698076 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 00:01:32.698904 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.698668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55284)
0807 00:01:32.698793 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.698843 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 00:01:32.699059 7791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:34243: Network error: Client connection negotiation failed: client connection to 127.8.204.124:34243: connect: Connection refused (error 111)
I20260807 00:01:32.700425 7803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:36473,127.8.204.125:43633,127.8.204.124:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:32.701435 7803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:32.702461 7803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:32.710927 7818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:32.710940 7810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:32.710932 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:32.711319 7803 server_base.cc:1034] running on GCE node
I20260807 00:01:32.711830 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:32.712918 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:32.714107 7803 hybrid_clock.cc:648] HybridClock initialized: now 1786060892714083 us; error 35 us; skew 500 ppm
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:32.716935 7803 init.cc:377] Logged in from keytab as kudu/127.8.204.124@KRBTEST.COM (short username kudu)
I20260807 00:01:32.717825 7803 webserver.cc:460] Webserver started at http://127.8.204.124:43911/ using document root <none> and password file <none>
I20260807 00:01:32.718323 7803 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:32.718381 7803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:32.718536 7803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:32.720479 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data/instance:
uuid: "9253435159bb4e3e91f586b49bfb964a"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "68c8bff54fc219458d71eef91a3f241d"
server_key_iv: "0a69e7dd740decfe2eb78b10d45ab7ac"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.720968 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal/instance:
uuid: "9253435159bb4e3e91f586b49bfb964a"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "68c8bff54fc219458d71eef91a3f241d"
server_key_iv: "0a69e7dd740decfe2eb78b10d45ab7ac"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.724531 7803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 00:01:32.726890 7826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:32.728061 7803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:32.728222 7803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "9253435159bb4e3e91f586b49bfb964a"
format_stamp: "Formatted at 2026-08-07 00:01:32 on dist-test-slave-39l8"
server_key: "68c8bff54fc219458d71eef91a3f241d"
server_key_iv: "0a69e7dd740decfe2eb78b10d45ab7ac"
server_key_version: "encryptionkey@0"
I20260807 00:01:32.728324 7803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:32.759894 7803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:32.763373 7803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:32.763587 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:32.770689 7803 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.124:34243
I20260807 00:01:32.771005 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.124:34243 every 8 connection(s)
I20260807 00:01:32.771781 7803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
I20260807 00:01:32.773525 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 7803
I20260807 00:01:32.773689 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060609240246-9009-0/minicluster-data/master-2/wal/instance
I20260807 00:01:32.774016 9009 external_mini_cluster.cc:1428] Setting key 42e295df65e8336fa75bc4d330150e37
I20260807 00:01:32.774945 7901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:32.779281 7901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.782491 7690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 3
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:32.790212 7791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } attempt: 3
I20260807 00:01:32.790637 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.775507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.8.204.124:34243)
0807 00:01:32.775928 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.775944 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.775979 (+ 35us) server_negotiation.cc:407] Connection header received
0807 00:01:32.776644 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.776661 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.776977 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.777257 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.778204 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.779276 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.780393 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.780907 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.781055 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.784182 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.784188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.784197 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.784223 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.786380 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.786994 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.787001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.787008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.787179 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.787533 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.787537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.787538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.788088 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.788314 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.789118 (+ 804us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.789496 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":146,"threads_started":1}
I20260807 00:01:32.793926 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.782953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55298)
0807 00:01:32.783096 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.783111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.783204 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.783734 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.783737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.783921 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.784010 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.784017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.785001 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.785005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.786146 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.786158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.786822 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.786826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.786907 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.787351 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.787369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.789436 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.792210 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.792213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.792214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.792497 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.792808 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.792812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.792814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.792905 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.793357 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.793361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.793457 (+ 96us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.793615 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.793764 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":291}
I20260807 00:01:32.794543 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.783208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.8.204.124:34243)
0807 00:01:32.783497 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.783508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.783523 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:32.783560 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.783566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.783611 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.783698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.784172 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.784894 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.786291 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.786654 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.787121 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.789604 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.789610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.789612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.789639 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.792108 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.792604 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.792607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.792608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.792692 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.793022 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.793025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.793026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.793241 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.794087 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.794194 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.794319 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":216,"spinlock_wait_cycles":2432,"thread_start_us":99,"threads_started":1}
I20260807 00:01:32.795361 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.781225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.8.204.126:36473)
0807 00:01:32.781408 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.781415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.781742 (+ 327us) server_negotiation.cc:407] Connection header received
0807 00:01:32.781885 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.781888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.781952 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.782048 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.782793 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.783484 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.784819 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.785669 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.785807 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.789692 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.789696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.789698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.789720 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.792930 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.793472 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.793477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.793480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.793563 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.793883 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.793886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.793888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.794109 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.794188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.795071 (+ 883us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.795192 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":50}
I20260807 00:01:32.796463 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.781162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:36473 (local address 127.0.0.1:60498)
0807 00:01:32.781609 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.781657 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.781775 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.782101 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.782105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.782416 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.782567 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.782572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.783639 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.783650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.784687 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.784699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.785980 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.786025 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.786398 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.787430 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.787448 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.789556 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.793040 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.793049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.793054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.793352 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.793680 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.793682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.793684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.793782 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.794479 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.794485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.794826 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.796024 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.796208 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":24,"thread_start_us":154,"threads_started":1}
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260807 00:01:32.798185 7901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.798305 7690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: Bootstrap starting.
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:32.801152 7690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:32.802121 7690 log.cc:824] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:32.804301 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.790699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55304)
0807 00:01:32.790979 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.790993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.791095 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.792451 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.792457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.792647 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.792772 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.792781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.793801 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.793807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.794634 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.794649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.795298 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.795302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.795391 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.795878 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.795898 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.797384 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.802530 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.802533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.802535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.802831 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.803202 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.803208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.803213 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.803323 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.803826 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.803829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.803920 (+ 91us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.804015 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.804153 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:32.804752 7690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: No bootstrap required, opened a new log
I20260807 00:01:32.805106 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.790977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.8.204.124:34243)
0807 00:01:32.791283 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.792223 (+ 940us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.792235 (+ 12us) server_negotiation.cc:407] Connection header received
0807 00:01:32.792269 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.792272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.792324 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.792419 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.792915 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.793676 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.794812 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.795147 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.795619 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.797522 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.797526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.797528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.797552 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.802381 (+ 4829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.802959 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.802982 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.802987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.803079 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.803472 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.803475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.803477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.803687 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.803779 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.804829 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.804944 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260807 00:01:32.805706 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.794007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.8.204.126:36473)
0807 00:01:32.794615 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.794627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.794642 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:32.794679 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.794682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.794745 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.795710 (+ 965us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.795798 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.796549 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.797652 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.797975 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.798074 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.800807 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.800810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.800813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.800836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.803213 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.803777 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.803781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.803785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.803876 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.804247 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.804252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.804255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.804459 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.805284 (+ 825us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.805365 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.805497 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":88,"threads_started":1}
I20260807 00:01:32.808876 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.798696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:43633 (local address 127.0.0.1:50984)
0807 00:01:32.798847 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.798864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.798942 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.799270 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.799273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.799453 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.799556 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.799565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.800714 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.800720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.801637 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.801653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.802800 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.802804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.802916 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.803401 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.803415 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.804852 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.807215 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.807218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.807220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.807462 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.807776 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.807779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.807781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.807877 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.808372 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.808375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.808470 (+ 95us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.808584 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.808715 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 00:01:32.808838 7791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: Bootstrap starting.
I20260807 00:01:32.808938 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.798944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.8.204.125:43633)
0807 00:01:32.799046 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.799050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.799063 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:01:32.799106 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.799109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.799145 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.799222 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.799734 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.800553 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.801847 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.802193 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.802301 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.805010 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.805014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.805017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.805045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.807081 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.807575 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.807578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.807580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.807669 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.808005 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.808009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.808010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.808260 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.808364 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.808650 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.808763 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 00:01:32.809573 7690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.809789 7690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e4015198904560b8dcbe6314761547, State: Initialized, Role: FOLLOWER
I20260807 00:01:32.809875 7901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:32.810139 7690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.811619 7791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:32.811712 7690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:32.812490 7791 log.cc:824] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:32.814684 7791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: No bootstrap required, opened a new log
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
Aug 07 00:01:32 dist-test-slave-39l8 krb5kdc[7568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260807 00:01:32.819890 7791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.820142 7791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8896bbbad44ecc9125515aa8d13b1c, State: Initialized, Role: FOLLOWER
I20260807 00:01:32.820497 7791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.820889 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.810551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.8.204.124:34243)
0807 00:01:32.810648 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.810651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.810929 (+ 278us) server_negotiation.cc:407] Connection header received
0807 00:01:32.811060 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.811063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.811096 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.811143 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.811594 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.812297 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.813613 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.813961 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.814054 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.817495 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.817498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.817501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.817522 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.819177 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.819658 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.819661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.819662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.819736 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.820009 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.820012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.820014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.820236 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.820323 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.820608 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.820725 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 00:01:32.816870 7583 master_runner.cc:186] Error getting master registration for 127.8.204.125:43633: OK, instance_id { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" instance_seqno: 1786060892534783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 00:01:32.821257 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.810366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:34243 (local address 127.0.0.1:55316)
0807 00:01:32.810867 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:32.810881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:32.810978 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:32.811254 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:32.811256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:32.811406 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:32.811479 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.811485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.812642 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.812646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.813460 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:32.813478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.814219 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.814222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:32.814292 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.814736 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:32.814815 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:32.817356 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:32.819294 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.819297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.819299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.819554 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.819830 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:32.819832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:32.819834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:32.819914 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:32.820380 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:32.820382 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:32.820456 (+ 74us) client_negotiation.cc:769] Sending connection context
0807 00:01:32.821010 (+ 554us) client_negotiation.cc:240] Negotiation successful
0807 00:01:32.821136 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444}
I20260807 00:01:32.821368 7583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 00:01:32.821894 7791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:32.822111 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.806879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.8.204.125:43633)
0807 00:01:32.809064 (+ 2185us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.809067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.809092 (+ 25us) server_negotiation.cc:407] Connection header received
0807 00:01:32.809125 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.809128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.809165 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.809238 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.810921 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.811579 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.812682 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.812987 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.813099 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.816828 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.816832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.816834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.816856 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.818857 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.819358 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.819361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.819363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.819446 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.819780 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.819786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.819788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.819989 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.821266 (+ 1277us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.821875 (+ 609us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.821983 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2133}
W20260807 00:01:32.823710 7929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:32.823776 7929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:32.824000 7914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.824115 7914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.824576 7901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: Bootstrap starting.
I20260807 00:01:32.825431 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:32.826773 7650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "16e4015198904560b8dcbe6314761547"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60508
I20260807 00:01:32.826905 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.826967 7650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 00:01:32.827023 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.827065 7650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:32.827299 7901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:32.828159 7901 log.cc:824] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:32.828326 7650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.829306 7650 leader_election.cc:288] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8896bbbad44ecc9125515aa8d13b1c (127.8.204.125:43633), 9253435159bb4e3e91f586b49bfb964a (127.8.204.124:34243)
I20260807 00:01:32.832046 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e4015198904560b8dcbe6314761547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a8896bbbad44ecc9125515aa8d13b1c"
I20260807 00:01:32.832309 7757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:32.833561 7757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16e4015198904560b8dcbe6314761547 in term 1.
I20260807 00:01:32.833982 7612 leader_election.cc:302] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e4015198904560b8dcbe6314761547, 7a8896bbbad44ecc9125515aa8d13b1c; no voters:
I20260807 00:01:32.834301 7914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:32.834532 7914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [term 1 LEADER]: Becoming Leader. State: Replica: 16e4015198904560b8dcbe6314761547, State: Running, Role: LEADER
I20260807 00:01:32.834930 7914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.829452 7701 master_runner.cc:186] Error getting master registration for 127.8.204.124:34243: OK, instance_id { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" instance_seqno: 1786060892768765 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 00:01:32.835142 7701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 00:01:32.837177 7901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: No bootstrap required, opened a new log
W20260807 00:01:32.837615 7949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:32.837678 7949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:32.844424 7901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.844930 7901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9253435159bb4e3e91f586b49bfb964a, State: Initialized, Role: FOLLOWER
I20260807 00:01:32.845547 7901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } }
I20260807 00:01:32.847664 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.831756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.8.204.124:34243)
0807 00:01:32.831880 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.831884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.836219 (+ 4335us) server_negotiation.cc:407] Connection header received
0807 00:01:32.836390 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.836393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.836440 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.836494 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.837516 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.838268 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.839625 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.839972 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.840076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.841986 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.841990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.841993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.842020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.844053 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.845745 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.845751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.845754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.845865 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.846281 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.846284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.846286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.846496 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.846663 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.847334 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.847465 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 00:01:32.848613 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e4015198904560b8dcbe6314761547. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.848757 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:32.849195 7966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:32.849434 7901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:32.850253 7867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e4015198904560b8dcbe6314761547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9253435159bb4e3e91f586b49bfb964a"
I20260807 00:01:32.850376 7966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:32.850502 7867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:32.851351 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.831576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.8.204.126:36473)
0807 00:01:32.831770 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.831775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.838791 (+ 7016us) server_negotiation.cc:407] Connection header received
0807 00:01:32.838838 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.838841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.838893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.839007 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.839867 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.840676 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.841846 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.842244 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.842352 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.846817 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.846822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.846826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.846856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.849086 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.849676 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.849679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.849681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.849765 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.850102 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.850104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.850106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.850341 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.851000 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.851078 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.851205 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 00:01:32.851967 7867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16e4015198904560b8dcbe6314761547 in term 1.
I20260807 00:01:32.854404 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:32.832768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.8.204.125:43633)
0807 00:01:32.832903 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.832907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.838004 (+ 5097us) server_negotiation.cc:407] Connection header received
0807 00:01:32.842948 (+ 4944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.842954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.843008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.843082 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.843933 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.844700 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.845779 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:32.846107 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:32.846187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:32.848206 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:32.848212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:32.848216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:32.848246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:32.850108 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.853016 (+ 2908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.853019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.853021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.853110 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:32.853424 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:32.853427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:32.853429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:32.853632 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:32.853765 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:32.854117 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 00:01:32.854253 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 00:01:32.856383 7977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:32.856482 7977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:32.856710 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.856827 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.860996 7757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [term 1 FOLLOWER]: Refusing update from remote peer 16e4015198904560b8dcbe6314761547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:32.861007 7867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [term 1 FOLLOWER]: Refusing update from remote peer 16e4015198904560b8dcbe6314761547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:32.861701 7960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:32.861862 7914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:32.866008 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e4015198904560b8dcbe6314761547. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.866148 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.866235 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e4015198904560b8dcbe6314761547. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.866349 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.868695 7914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.868819 7914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:32.869151 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.869309 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.869457 7978 mvcc.cc:205] Tried to move back new op lower bound from 7315705417150664704 to 7315705417055703040. Current Snapshot: MvccSnapshot[applied={T|T < 7315705417150664704}]
I20260807 00:01:32.871083 7984 mvcc.cc:205] Tried to move back new op lower bound from 7315705417150664704 to 7315705417055703040. Current Snapshot: MvccSnapshot[applied={T|T < 7315705417150664704}]
I20260807 00:01:32.875521 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.875633 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547 [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:32.876698 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16e4015198904560b8dcbe6314761547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e4015198904560b8dcbe6314761547" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 36473 } } peers { permanent_uuid: "7a8896bbbad44ecc9125515aa8d13b1c" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 43633 } } peers { permanent_uuid: "9253435159bb4e3e91f586b49bfb964a" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 34243 } } }
I20260807 00:01:32.876808 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:32.877405 7983 mvcc.cc:205] Tried to move back new op lower bound from 7315705417150664704 to 7315705417055703040. Current Snapshot: MvccSnapshot[applied={T|T < 7315705417150664704}]
I20260807 00:01:32.888156 7966 catalog_manager.cc:1347] Generated new cluster ID: d5308f7f824e44bcbfd9d27cd53cb329
I20260807 00:01:32.888227 7966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:32.909667 7966 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:32.910945 7966 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:32.941433 7966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e4015198904560b8dcbe6314761547: Generated new TSK 0
I20260807 00:01:32.942356 7966 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 00:01:32.976691 7912 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.973472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.8.204.126:36473)
0807 00:01:32.974282 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.974289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.975596 (+ 1307us) server_negotiation.cc:407] Connection header received
0807 00:01:32.975803 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.975807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.975857 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.975911 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.976551 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60522: BlockingRecv error: recv got EOF from 127.0.0.1:60522 (error 108)
Metrics: {"server-negotiator.queue_time_us":712}
W20260807 00:01:32.978366 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.974243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.8.204.125:43633)
0807 00:01:32.974356 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.974360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.977069 (+ 2709us) server_negotiation.cc:407] Connection header received
0807 00:01:32.977397 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.977400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.977452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.977516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.978243 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51006: BlockingRecv error: recv got EOF from 127.0.0.1:51006 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 00:01:32.979554 7911 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.974966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.8.204.124:34243)
0807 00:01:32.975077 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.975081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.977760 (+ 2679us) server_negotiation.cc:407] Connection header received
0807 00:01:32.978819 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.978822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.978872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.978924 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.979402 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55336: BlockingRecv error: recv got EOF from 127.0.0.1:55336 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260807 00:01:32.984555 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.983117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.8.204.125:43633)
0807 00:01:32.983216 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.983220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.983655 (+ 435us) server_negotiation.cc:407] Connection header received
0807 00:01:32.983858 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.983862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.983933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.983982 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.984439 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51022: BlockingRecv error: recv got EOF from 127.0.0.1:51022 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260807 00:01:32.989480 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.987993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.8.204.125:43633)
0807 00:01:32.988110 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.988114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.988638 (+ 524us) server_negotiation.cc:407] Connection header received
0807 00:01:32.988816 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.988820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.988870 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.988918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.989388 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51032: BlockingRecv error: recv got EOF from 127.0.0.1:51032 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260807 00:01:32.995039 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:32.993460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.8.204.125:43633)
0807 00:01:32.993562 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:32.993566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:32.994076 (+ 510us) server_negotiation.cc:407] Connection header received
0807 00:01:32.994274 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:32.994277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:32.994324 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:32.994372 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:32.994950 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 00:01:33.002776 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.001098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.8.204.125:43633)
0807 00:01:33.001228 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.001235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.001773 (+ 538us) server_negotiation.cc:407] Connection header received
0807 00:01:33.001963 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.001966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.002022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.002072 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.002613 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51046: BlockingRecv error: recv got EOF from 127.0.0.1:51046 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 00:01:33.014822 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.013084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.8.204.125:43633)
0807 00:01:33.013196 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.013200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.013776 (+ 576us) server_negotiation.cc:407] Connection header received
0807 00:01:33.013991 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.013994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.014055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.014110 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.014681 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51054: BlockingRecv error: recv got EOF from 127.0.0.1:51054 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260807 00:01:33.035171 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.033395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.8.204.125:43633)
0807 00:01:33.033518 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.033522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.034091 (+ 569us) server_negotiation.cc:407] Connection header received
0807 00:01:33.034299 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.034303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.034364 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.034428 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.035060 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51058: BlockingRecv error: recv got EOF from 127.0.0.1:51058 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 00:01:33.070659 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.069604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.8.204.125:43633)
0807 00:01:33.069741 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.069746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.069828 (+ 82us) server_negotiation.cc:407] Connection header received
0807 00:01:33.069959 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.069963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.070020 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.070115 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.070530 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51066: BlockingRecv error: recv got EOF from 127.0.0.1:51066 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 00:01:33.138072 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.136898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.8.204.125:43633)
0807 00:01:33.137105 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.137108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.137171 (+ 63us) server_negotiation.cc:407] Connection header received
0807 00:01:33.137320 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.137323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.137374 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.137448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.137917 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51080: BlockingRecv error: recv got EOF from 127.0.0.1:51080 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260807 00:01:33.269644 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.268385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.8.204.125:43633)
0807 00:01:33.268599 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.268602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.268622 (+ 20us) server_negotiation.cc:407] Connection header received
0807 00:01:33.268781 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.268784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.268837 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.268918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.269491 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51086: BlockingRecv error: recv got EOF from 127.0.0.1:51086 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260807 00:01:33.528805 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:33.527830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.8.204.125:43633)
0807 00:01:33.527959 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:33.527963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:33.528035 (+ 72us) server_negotiation.cc:407] Connection header received
0807 00:01:33.528161 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:33.528163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:33.528214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:33.528287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:33.528707 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51090: BlockingRecv error: recv got EOF from 127.0.0.1:51090 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 00:01:33.839213 7949 catalog_manager.cc:1259] Loaded cluster ID: d5308f7f824e44bcbfd9d27cd53cb329
I20260807 00:01:33.839310 7949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: loading cluster ID for follower catalog manager: success
I20260807 00:01:33.840983 7949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: acquiring CA information for follower catalog manager: success
I20260807 00:01:33.842147 7949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a8896bbbad44ecc9125515aa8d13b1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 00:01:33.857828 7977 catalog_manager.cc:1259] Loaded cluster ID: d5308f7f824e44bcbfd9d27cd53cb329
I20260807 00:01:33.858037 7977 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: loading cluster ID for follower catalog manager: success
I20260807 00:01:33.859756 7977 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: acquiring CA information for follower catalog manager: success
I20260807 00:01:33.860944 7977 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9253435159bb4e3e91f586b49bfb964a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 00:01:34.047233 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:34.043003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.8.204.125:43633)
0807 00:01:34.043245 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:34.043249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:34.043265 (+ 16us) server_negotiation.cc:407] Connection header received
0807 00:01:34.043479 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:34.043483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:34.043629 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:34.043740 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:34.044074 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.044665 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.045558 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.045828 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.045925 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:34.046279 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:34.046845 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:34.046956 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:34.047041 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 00:01:34.047099 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260807 00:01:34.054845 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:34.051021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.8.204.124:34243)
0807 00:01:34.051250 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:34.051254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:34.051311 (+ 57us) server_negotiation.cc:407] Connection header received
0807 00:01:34.051415 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:34.051418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:34.051539 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:34.051649 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:34.051854 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.052441 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.053307 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.053545 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.053639 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:34.053896 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:34.054312 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:34.054409 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:34.054563 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 00:01:34.054647 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260807 00:01:34.061882 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:34.058010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.8.204.124:34243)
0807 00:01:34.058122 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:34.058128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:34.058303 (+ 175us) server_negotiation.cc:407] Connection header received
0807 00:01:34.058498 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:34.058501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:34.058552 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:34.058606 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:34.059010 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.059787 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.060851 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.061061 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.061170 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:34.061431 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:34.061498 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:34.061580 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:34.061672 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 00:01:34.061728 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 00:01:34.062752 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:34.058263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.8.204.125:43633)
0807 00:01:34.058390 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:34.058394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:34.058673 (+ 279us) server_negotiation.cc:407] Connection header received
0807 00:01:34.058804 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:34.058810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:34.058871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:34.058936 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:34.059240 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.059739 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.061220 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.061549 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.061630 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:34.062229 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:34.062308 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:34.062356 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:34.062542 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 00:01:34.062605 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 00:01:34.063154 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:34.057894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.8.204.126:36473)
0807 00:01:34.058417 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:34.058426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:34.058443 (+ 17us) server_negotiation.cc:407] Connection header received
0807 00:01:34.058545 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:34.058554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:34.058712 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:34.058861 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:34.059067 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.059862 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.061209 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:34.061557 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:34.061645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:34.062272 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:34.062638 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:34.062764 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:34.062947 (+ 183us) server_negotiation.cc:299] Negotiation successful
0807 00:01:34.063012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":91,"threads_started":1}
I20260807 00:01:34.071103 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7583
I20260807 00:01:34.085036 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7701
I20260807 00:01:34.096387 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 7803
2026-08-07T00:01:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3292 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 00:01:34 dist-test-slave-39l8 krb5kdc[8108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 00:01:34 dist-test-slave-39l8 krb5kdc[8108](info): set up 2 sockets
Aug 07 00:01:34 dist-test-slave-39l8 krb5kdc[8108](info): commencing operation
krb5kdc: starting...
W20260807 00:01:35.294068 9009 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/test-user.keytab.
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T00:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T00:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
Entry for principal kudu/127.8.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab.
I20260807 00:01:35.417703 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37491
--webserver_interface=127.8.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:35.528286 8127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:35.528623 8127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:35.528697 8127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:35.528774 8127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:35.532845 8127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:35.532933 8127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:35.532974 8127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:35.533030 8127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:35.533072 8127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:35.538249 8127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.126.keytab
--principal=kudu/127.8.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.126:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:35.539382 8127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:35.540416 8127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:35.548700 8133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:35.548799 8127 server_base.cc:1034] running on GCE node
W20260807 00:01:35.548732 8132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:35.548746 8135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:35.549444 8127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:35.550465 8127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:35.551630 8127 hybrid_clock.cc:648] HybridClock initialized: now 1786060895551613 us; error 28 us; skew 500 ppm
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:35.554320 8127 init.cc:377] Logged in from keytab as kudu/127.8.204.126@KRBTEST.COM (short username kudu)
I20260807 00:01:35.555239 8127 webserver.cc:460] Webserver started at http://127.8.204.126:44693/ using document root <none> and password file <none>
I20260807 00:01:35.555753 8127 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:35.555811 8127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:35.555970 8127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:35.557924 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data/instance:
uuid: "3c06941586994d82992da5272faec75c"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "8074686d8dea4f7f38276ac8c373bc9d"
server_key_iv: "6be58004fca02ec4f660884df4d8616b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.558435 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance:
uuid: "3c06941586994d82992da5272faec75c"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "8074686d8dea4f7f38276ac8c373bc9d"
server_key_iv: "6be58004fca02ec4f660884df4d8616b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.562071 8127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 00:01:35.564468 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:35.565661 8127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 00:01:35.565805 8127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal
uuid: "3c06941586994d82992da5272faec75c"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "8074686d8dea4f7f38276ac8c373bc9d"
server_key_iv: "6be58004fca02ec4f660884df4d8616b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.565918 8127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:35.608786 8127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:35.612119 8127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:35.612330 8127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:35.619695 8127 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.126:37491
I20260807 00:01:35.620083 8222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.126:37491 every 8 connection(s)
I20260807 00:01:35.620687 8127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/data/info.pb
I20260807 00:01:35.623744 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:35.624230 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 8127
I20260807 00:01:35.624372 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-0/wal/instance
I20260807 00:01:35.624734 9009 external_mini_cluster.cc:1428] Setting key aa5e4247a7c06555120d40e2e95996b7
I20260807 00:01:35.628943 8223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.125@KRBTEST.COM" created.
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:35.645632 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.632355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.8.204.126:37491)
0807 00:01:35.632662 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:35.632673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:35.632711 (+ 38us) server_negotiation.cc:407] Connection header received
0807 00:01:35.633758 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:35.633763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:35.633967 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:35.634183 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:35.635126 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.636095 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.637307 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.637976 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.638165 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.640805 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:35.640811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:35.640820 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:35.640840 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:35.642513 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.643029 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.643041 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.643042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.643141 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.643466 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.643469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.643471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.643835 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:35.643974 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:35.644360 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 00:01:35.644620 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":193,"thread_start_us":120,"threads_started":1}
I20260807 00:01:35.645632 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.631556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37491 (local address 127.0.0.1:52616)
0807 00:01:35.632135 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.632195 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:35.633393 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:35.634218 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:35.634224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:35.634598 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:35.634914 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.634929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.636209 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.636212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.637182 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.637193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.638130 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.638133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.638212 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.638956 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:35.638984 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:35.640656 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:35.642626 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.642636 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.642646 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.642919 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.643248 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.643252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.643253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.643367 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.643960 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:35.643966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:35.644103 (+ 137us) client_negotiation.cc:769] Sending connection context
0807 00:01:35.644271 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 00:01:35.644669 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":126,"thread_start_us":109,"threads_started":1}
I20260807 00:01:35.648218 8223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:35.648974 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.648719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51034)
0807 00:01:35.648842 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.648910 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 00:01:35.649305 8151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
I20260807 00:01:35.650238 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.650044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51038)
0807 00:01:35.650142 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.650185 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 00:01:35.650410 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:37705: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
Entry for principal kudu/127.8.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab.
I20260807 00:01:35.651767 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:37705
--webserver_interface=127.8.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 00:01:35.699730 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } attempt: 1
I20260807 00:01:35.700645 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.700337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51052)
0807 00:01:35.700491 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.700563 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 00:01:35.701427 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.701206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51062)
0807 00:01:35.701317 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.701371 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 00:01:35.701593 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:37705: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
W20260807 00:01:35.764099 8234 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:35.764353 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:35.764379 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:35.764487 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:35.768486 8234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:35.768534 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:35.768554 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:35.768571 8234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:35.768625 8234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:35.769738 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } attempt: 2
I20260807 00:01:35.770606 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.770278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51072)
0807 00:01:35.770428 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.770492 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 00:01:35.771406 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.771214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51076)
0807 00:01:35.771311 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.771354 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 00:01:35.771551 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.125:37705: Network error: Client connection negotiation failed: client connection to 127.8.204.125:37705: connect: Connection refused (error 111)
I20260807 00:01:35.774329 8234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.125.keytab
--principal=kudu/127.8.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.125:37705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:35.775364 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:35.776288 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:35.784559 8242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:35.784569 8241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:35.784566 8244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:35.784837 8234 server_base.cc:1034] running on GCE node
I20260807 00:01:35.785465 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:35.786561 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:35.787758 8234 hybrid_clock.cc:648] HybridClock initialized: now 1786060895787742 us; error 36 us; skew 500 ppm
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:35.790403 8234 init.cc:377] Logged in from keytab as kudu/127.8.204.125@KRBTEST.COM (short username kudu)
I20260807 00:01:35.791350 8234 webserver.cc:460] Webserver started at http://127.8.204.125:36119/ using document root <none> and password file <none>
I20260807 00:01:35.791927 8234 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:35.792058 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:35.792299 8234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:35.794286 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data/instance:
uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "c9b7f56dfb3c77c74fbc8d5823730c79"
server_key_iv: "28d1c318d298be73793480489a87b45b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.794914 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal/instance:
uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "c9b7f56dfb3c77c74fbc8d5823730c79"
server_key_iv: "28d1c318d298be73793480489a87b45b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.800375 8234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260807 00:01:35.802801 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:35.804003 8234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 00:01:35.804147 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal
uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a"
format_stamp: "Formatted at 2026-08-07 00:01:35 on dist-test-slave-39l8"
server_key: "c9b7f56dfb3c77c74fbc8d5823730c79"
server_key_iv: "28d1c318d298be73793480489a87b45b"
server_key_version: "encryptionkey@0"
I20260807 00:01:35.804267 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:35.832154 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:35.835055 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:35.835211 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:35.841948 8234 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.125:37705
I20260807 00:01:35.842239 8326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.125:37705 every 8 connection(s)
I20260807 00:01:35.842967 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/data/info.pb
I20260807 00:01:35.845674 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:35.848816 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 8234
I20260807 00:01:35.848927 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-1/wal/instance
I20260807 00:01:35.849215 9009 external_mini_cluster.cc:1428] Setting key e39ddf47d1165ded6596a77209592653
I20260807 00:01:35.849485 8328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.124@KRBTEST.COM" created.
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:35.862829 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } attempt: 3
I20260807 00:01:35.866307 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.851390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.8.204.126:37491)
0807 00:01:35.851571 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:35.851576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:35.852519 (+ 943us) server_negotiation.cc:407] Connection header received
0807 00:01:35.853469 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:35.853472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:35.853537 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:35.853640 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:35.854903 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.855430 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.856545 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.856782 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.856900 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.861140 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:35.861143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:35.861146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:35.861183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:35.863026 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.863647 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.863651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.863654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.863758 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.864178 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.864180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.864182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.864354 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:35.864464 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:35.865929 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0807 00:01:35.866080 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 00:01:35.866334 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.851796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37491 (local address 127.0.0.1:52632)
0807 00:01:35.852321 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.852380 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:35.853263 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:35.853808 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:35.853814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:35.854235 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:35.854743 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.854771 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.855560 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.855564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.856418 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.856427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.856940 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.856945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.857122 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.857720 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:35.857747 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:35.860970 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:35.863178 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.863192 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.863207 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.863540 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.863875 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.863881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.863886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.864054 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.864465 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:35.864473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:35.864809 (+ 336us) client_negotiation.cc:769] Sending connection context
0807 00:01:35.865030 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 00:01:35.865355 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260807 00:01:35.868479 8328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:35.873534 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.863323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51078)
0807 00:01:35.863438 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.863454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:35.863536 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:35.864646 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:35.864649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:35.864824 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:35.864939 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.864947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.866244 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.866248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.866949 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.866960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.867663 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.867666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.867775 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.868243 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:35.868263 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:35.869737 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:35.871685 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.871688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.871692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.871966 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.872275 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.872279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.872281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.872384 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.872971 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:35.872975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:35.873078 (+ 103us) client_negotiation.cc:769] Sending connection context
0807 00:01:35.873206 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 00:01:35.873359 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 00:01:35.873734 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.863759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.8.204.125:37705)
0807 00:01:35.864156 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:35.864170 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:35.864198 (+ 28us) server_negotiation.cc:407] Connection header received
0807 00:01:35.864247 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:35.864255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:35.864460 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:35.864651 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:35.865064 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.866103 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.867071 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.867545 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.867665 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.869887 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:35.869896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:35.869910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:35.869939 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:35.871522 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.872100 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.872104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.872105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.872175 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.872468 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.872471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.872472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.872791 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:35.872995 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:35.873346 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 00:01:35.873512 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":161,"threads_started":1}
Aug 07 00:01:35 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:35.875924 8223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
Entry for principal kudu/127.8.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab.
I20260807 00:01:35.876760 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.876454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:44970)
0807 00:01:35.876606 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.876671 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 00:01:35.877617 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.877331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:44986)
0807 00:01:35.877499 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.877552 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260807 00:01:35.877584 9009 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
/tmp/dist-test-taskcJxSMk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:43863
--webserver_interface=127.8.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.84:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 00:01:35.877753 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:35.879288 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.868992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51092)
0807 00:01:35.869141 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.869157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:35.869238 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:35.870248 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:35.870252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:35.870428 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:35.870530 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.870535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.871696 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.871700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.872235 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:35.872276 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.872884 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.872889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:35.873037 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.873475 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:35.873489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:35.875625 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:35.877823 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.877827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.877829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.878039 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.878274 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:35.878277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:35.878279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:35.878360 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:35.878764 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:35.878767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:35.878870 (+ 103us) client_negotiation.cc:769] Sending connection context
0807 00:01:35.878993 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 00:01:35.879131 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 00:01:35.879343 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.869511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.8.204.125:37705)
0807 00:01:35.869981 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:35.869989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:35.870003 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:01:35.870041 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:35.870045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:35.870099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:35.870221 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:35.870934 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.871574 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.872371 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:35.872724 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:35.872853 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:35.875801 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:35.875805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:35.875807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:35.875831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:35.877696 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.878119 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.878120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.878121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.878174 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:35.878451 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:35.878455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:35.878457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:35.878621 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:35.878781 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:35.879047 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 00:01:35.879196 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":377,"thread_start_us":132,"threads_started":1}
I20260807 00:01:35.880015 8328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:35.880681 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.880453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:44990)
0807 00:01:35.880565 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.880630 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 00:01:35.880906 8259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:35.881750 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.881552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45000)
0807 00:01:35.881646 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.881693 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 00:01:35.881960 8328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:35.908907 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 1
I20260807 00:01:35.909807 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.909450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45014)
0807 00:01:35.909641 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.909715 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 00:01:35.910591 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.910338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45030)
0807 00:01:35.910459 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.910523 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 00:01:35.910761 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:35.931205 8328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 1
I20260807 00:01:35.932118 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.931768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45036)
0807 00:01:35.931918 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.932017 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 00:01:35.932859 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.932633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45048)
0807 00:01:35.932755 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.932807 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 00:01:35.932987 8328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:35.985941 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 2
I20260807 00:01:35.986908 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.986542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45050)
0807 00:01:35.986736 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.986812 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 00:01:35.987655 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:35.987399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45060)
0807 00:01:35.987524 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:35.987590 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 00:01:35.987790 8223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
W20260807 00:01:35.993520 8338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 00:01:35.993769 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 00:01:35.993793 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 00:01:35.993836 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 00:01:35.997807 8338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 00:01:35.997856 8338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 00:01:35.997876 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 00:01:35.997892 8338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 00:01:35.997907 8338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 00:01:36.001137 8328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 2
I20260807 00:01:36.001883 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.001589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45066)
0807 00:01:36.001740 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.001800 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 00:01:36.002605 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.002400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45068)
0807 00:01:36.002506 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.002550 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 00:01:36.002767 8328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.124:43863: Network error: Client connection negotiation failed: client connection to 127.8.204.124:43863: connect: Connection refused (error 111)
I20260807 00:01:36.003180 8338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.84:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.126:37491,127.8.204.125:37705,127.8.204.124:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/krb5kdc/kudu_127.8.204.124.keytab
--principal=kudu/127.8.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.124:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260807 00:01:36.004210 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 00:01:36.005244 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 00:01:36.014170 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:36.014206 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 00:01:36.014566 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 00:01:36.015020 8338 server_base.cc:1034] running on GCE node
I20260807 00:01:36.015753 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 00:01:36.017114 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 00:01:36.018272 8338 hybrid_clock.cc:648] HybridClock initialized: now 1786060896018224 us; error 66 us; skew 500 ppm
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 00:01:36.021250 8338 init.cc:377] Logged in from keytab as kudu/127.8.204.124@KRBTEST.COM (short username kudu)
I20260807 00:01:36.022071 8338 webserver.cc:460] Webserver started at http://127.8.204.124:35919/ using document root <none> and password file <none>
I20260807 00:01:36.022632 8338 fs_manager.cc:359] Metadata directory not provided
I20260807 00:01:36.022747 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 00:01:36.022989 8338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 00:01:36.024978 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data/instance:
uuid: "3db68f4b9a3840a4bd8824fee4e83ed4"
format_stamp: "Formatted at 2026-08-07 00:01:36 on dist-test-slave-39l8"
server_key: "062742d539cbb872e559572b7b1e0daa"
server_key_iv: "dfcb99336d61db93ea19a4a3bc58792d"
server_key_version: "encryptionkey@0"
I20260807 00:01:36.025557 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal/instance:
uuid: "3db68f4b9a3840a4bd8824fee4e83ed4"
format_stamp: "Formatted at 2026-08-07 00:01:36 on dist-test-slave-39l8"
server_key: "062742d539cbb872e559572b7b1e0daa"
server_key_iv: "dfcb99336d61db93ea19a4a3bc58792d"
server_key_version: "encryptionkey@0"
I20260807 00:01:36.029173 8338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 00:01:36.031661 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 00:01:36.032974 8338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 00:01:36.033118 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal
uuid: "3db68f4b9a3840a4bd8824fee4e83ed4"
format_stamp: "Formatted at 2026-08-07 00:01:36 on dist-test-slave-39l8"
server_key: "062742d539cbb872e559572b7b1e0daa"
server_key_iv: "dfcb99336d61db93ea19a4a3bc58792d"
server_key_version: "encryptionkey@0"
I20260807 00:01:36.033236 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 00:01:36.045706 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 00:01:36.048969 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 00:01:36.049173 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 00:01:36.056075 8338 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.124:43863
I20260807 00:01:36.056372 8434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.124:43863 every 8 connection(s)
I20260807 00:01:36.057022 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/data/info.pb
I20260807 00:01:36.059902 8435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 00:01:36.063695 8435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
I20260807 00:01:36.065718 9009 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu as pid 8338
I20260807 00:01:36.065868 9009 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJxSMk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060609240246-9009-0/minicluster-data/master-2/wal/instance
I20260807 00:01:36.066125 9009 external_mini_cluster.cc:1428] Setting key 2c0d68ff13e19258cf737d0151342780
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260807 00:01:36.078927 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.065582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.8.204.126:37491)
0807 00:01:36.065746 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.065752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.066705 (+ 953us) server_negotiation.cc:407] Connection header received
0807 00:01:36.067800 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.067804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.067854 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.067946 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.069103 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.069959 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.071377 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.071724 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.071831 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.074754 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.074758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.074761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.074796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.076506 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.077175 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.077178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.077179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.077256 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.077580 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.077583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.077585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.077819 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.077948 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.078577 (+ 629us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.078709 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 00:01:36.079834 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.065990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.126:37491 (local address 127.0.0.1:52646)
0807 00:01:36.066540 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.066583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:36.067601 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:36.068046 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:36.068052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:36.068549 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:36.068952 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.068976 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.070085 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.070088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.071257 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.071269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.071869 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.071872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.072040 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.072737 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:36.072765 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:36.074514 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:36.076622 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.076630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.076638 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.077074 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.077344 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.077346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.077347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.077472 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.077955 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:36.077960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:36.078254 (+ 294us) client_negotiation.cc:769] Sending connection context
0807 00:01:36.078462 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 00:01:36.078821 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":7,"thread_start_us":152,"threads_started":1}
I20260807 00:01:36.079835 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.066878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.8.204.124:43863)
0807 00:01:36.067278 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.067290 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.067332 (+ 42us) server_negotiation.cc:407] Connection header received
0807 00:01:36.067613 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.067637 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.067886 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.068074 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.068971 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.070262 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.071433 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.072006 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.072135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.075226 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.075231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.075241 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.075262 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.077068 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.077719 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.077724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.077725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.077792 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.078100 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.078104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.078105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.078370 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.078558 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.078928 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.079060 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260807 00:01:36.081187 8435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
I20260807 00:01:36.086941 8223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 3
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.126@KRBTEST.COM
I20260807 00:01:36.090771 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.081698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.125:37705 (local address 127.0.0.1:51106)
0807 00:01:36.081806 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.081819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:36.081892 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:36.082149 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:36.082153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:36.082319 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:36.082412 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.082416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.083531 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.083534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.084306 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.084317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.084874 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.084877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.084948 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.085419 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:36.085435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:36.086981 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:36.089012 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.089015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.089017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.089212 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.089501 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.089507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.089510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.089608 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.090151 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:36.090154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:36.090274 (+ 120us) client_negotiation.cc:769] Sending connection context
0807 00:01:36.090394 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 00:01:36.090516 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 00:01:36.090850 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.081809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.8.204.125:37705)
0807 00:01:36.081933 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.081940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.081955 (+ 15us) server_negotiation.cc:407] Connection header received
0807 00:01:36.082005 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.082007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.082047 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.082135 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.082560 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.083414 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.084427 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.084757 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.084871 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.087195 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.087201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.087205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.087236 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.088892 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.089323 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.089327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.089329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.089423 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.089736 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.089742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.089746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.089972 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.090067 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.090472 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.090619 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 00:01:36.092228 8435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.126@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260807 00:01:36.093295 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.082541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.8.204.126:37491)
0807 00:01:36.082867 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.082873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.082885 (+ 12us) server_negotiation.cc:407] Connection header received
0807 00:01:36.082917 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.082922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.082964 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.083042 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.083736 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.084450 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.085452 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.085784 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.085918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.088971 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.088975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.088977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.089002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.090862 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.091346 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.091349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.091351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.091437 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.091751 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.091754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.091756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.092203 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.092504 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.092839 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.092965 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260807 00:01:36.093921 8328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } attempt: 3
I20260807 00:01:36.096591 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.087326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45090)
0807 00:01:36.087421 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.087436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:36.087535 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:36.088177 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:36.088180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:36.088364 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:36.088475 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.088493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.089463 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.089469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.090115 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.090125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.090700 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.090704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.090804 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.091292 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:36.091306 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:36.092699 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:36.094952 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.094955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.094957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.095202 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.095537 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.095540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.095542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.095636 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.096121 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:36.096125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:36.096231 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 00:01:36.096315 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 00:01:36.096447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260807 00:01:36.097095 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.087378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.8.204.124:43863)
0807 00:01:36.087523 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.087529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.087543 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:01:36.087639 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.087645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.088056 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.088157 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.088625 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.089345 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.090239 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.090555 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.091220 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.092842 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.092846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.092848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.092874 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.094831 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.095336 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.095342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.095346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.095428 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.095753 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.095756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.095758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.095982 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.096776 (+ 794us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.096850 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.096956 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.124@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.125@KRBTEST.COM
Aug 07 00:01:36 dist-test-slave-39l8 krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.125@KRBTEST.COM for kudu/127.8.204.124@KRBTEST.COM
I20260807 00:01:36.102046 8223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: Bootstrap starting.
I20260807 00:01:36.102569 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.092722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45102)
0807 00:01:36.093518 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.093532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:36.093630 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:36.094124 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:36.094127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:36.094371 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:36.094462 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.094468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.095817 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.095823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.096311 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.096319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.096830 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.096833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.096899 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.097524 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:36.097538 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:36.098887 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:36.101047 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.101050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.101052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.101282 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.101571 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.101573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.101575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.101650 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.102082 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:36.102085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:36.102175 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 00:01:36.102267 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 00:01:36.102415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717}
I20260807 00:01:36.103207 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.092848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.8.204.124:43863)
0807 00:01:36.093879 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.093903 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.093921 (+ 18us) server_negotiation.cc:407] Connection header received
0807 00:01:36.093958 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.093960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.093999 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.094565 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.094652 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.095737 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.096430 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.096708 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.097091 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.099018 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.099021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.099024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.099044 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.100934 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.101390 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.101392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.101394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.101469 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.101743 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.101745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.101747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.101972 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.102773 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.102869 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.102995 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":136,"threads_started":1}
I20260807 00:01:36.105674 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.095079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.8.204.125:37705)
0807 00:01:36.095184 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.095199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.095362 (+ 163us) server_negotiation.cc:407] Connection header received
0807 00:01:36.095537 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.095541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.095580 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.095624 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.096471 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.097143 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.098402 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.098760 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.098826 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.101898 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.101901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.101904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.101930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.103670 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.104172 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.104176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.104177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.104274 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.104607 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.104611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.104612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.104818 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.105037 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.105408 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.105534 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 00:01:36.106107 8223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:36.106745 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.094303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.124:43863 (local address 127.0.0.1:45112)
0807 00:01:36.094449 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 00:01:36.094464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 00:01:36.094544 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 00:01:36.097481 (+ 2937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 00:01:36.097485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 00:01:36.097656 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0807 00:01:36.097754 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.097761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.098766 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.098769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.099518 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 00:01:36.099528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.100246 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.100251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 00:01:36.100362 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.100816 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 00:01:36.100831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 00:01:36.102329 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 00:01:36.104640 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.104644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.104646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.104846 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.105161 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 00:01:36.105163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 00:01:36.105165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 00:01:36.105256 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 00:01:36.106180 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 00:01:36.106186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 00:01:36.106285 (+ 99us) client_negotiation.cc:769] Sending connection context
0807 00:01:36.106408 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 00:01:36.106554 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 00:01:36.107111 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.094759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.8.204.124:43863)
0807 00:01:36.097236 (+ 2477us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.097242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.097255 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:01:36.097291 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.097295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.097345 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.097862 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.097939 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.098606 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.099698 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.100093 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.100580 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.102481 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.102487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.102491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.102522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.104511 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.104983 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.104986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.104988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.105070 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.105480 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.105484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.105488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.105648 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.106238 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.106558 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.106655 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2414,"thread_start_us":110,"threads_started":1}
I20260807 00:01:36.108217 8223 log.cc:824] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:36.109862 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: Bootstrap starting.
I20260807 00:01:36.109915 8435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: Bootstrap starting.
I20260807 00:01:36.110532 8223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: No bootstrap required, opened a new log
I20260807 00:01:36.112135 8328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:36.112363 8435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: Neither blocks nor log segments found. Creating new log.
I20260807 00:01:36.112902 8328 log.cc:824] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:36.113130 8435 log.cc:824] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: Log is configured to *not* fsync() on all Append() calls
I20260807 00:01:36.114850 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: No bootstrap required, opened a new log
I20260807 00:01:36.114862 8223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.115125 8223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c06941586994d82992da5272faec75c, State: Initialized, Role: FOLLOWER
I20260807 00:01:36.115532 9009 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 00:01:36.115469 8223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.115978 8445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.116197 8445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.116690 8186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3c06941586994d82992da5272faec75c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52648
I20260807 00:01:36.116873 8223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:36.116901 8186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 00:01:36.116997 8186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:36.118384 8186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.118405 8328 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.118642 8328 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bad300f5ef4bbaaf20d86cf9aeb79a, State: Initialized, Role: FOLLOWER
I20260807 00:01:36.119019 8328 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.119477 8186 leader_election.cc:288] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [CANDIDATE]: Term 1 election: Requested vote from peers a3bad300f5ef4bbaaf20d86cf9aeb79a (127.8.204.125:37705), 3db68f4b9a3840a4bd8824fee4e83ed4 (127.8.204.124:43863)
I20260807 00:01:36.120527 8328 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:36.121719 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c06941586994d82992da5272faec75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a"
I20260807 00:01:36.121941 8290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:36.123415 8290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c06941586994d82992da5272faec75c in term 1.
I20260807 00:01:36.123560 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.123701 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: This master's current role is: FOLLOWER
W20260807 00:01:36.124943 8460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:36.125078 8460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:36.125381 8435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: No bootstrap required, opened a new log
I20260807 00:01:36.121776 8127 master_runner.cc:186] Error getting master registration for 127.8.204.125:37705: OK, instance_id { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" instance_seqno: 1786060895840174 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 00:01:36.125666 8127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 00:01:36.125993 8151 leader_election.cc:302] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c06941586994d82992da5272faec75c, a3bad300f5ef4bbaaf20d86cf9aeb79a; no voters:
I20260807 00:01:36.126291 8445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 00:01:36.126497 8445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [term 1 LEADER]: Becoming Leader. State: Replica: 3c06941586994d82992da5272faec75c, State: Running, Role: LEADER
I20260807 00:01:36.126848 8445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
W20260807 00:01:36.128052 8480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:36.128127 8480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:36.128859 8445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c06941586994d82992da5272faec75c. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.128989 8445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:36.129405 8435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.129585 8435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db68f4b9a3840a4bd8824fee4e83ed4, State: Initialized, Role: FOLLOWER
I20260807 00:01:36.129813 8435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } }
I20260807 00:01:36.130196 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.130326 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.131124 8435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 00:01:36.133570 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c06941586994d82992da5272faec75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4"
I20260807 00:01:36.133797 8399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 00:01:36.135253 8399 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c06941586994d82992da5272faec75c in term 1.
W20260807 00:01:36.137969 8511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 00:01:36.138042 8511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 00:01:36.137974 8234 master_runner.cc:186] Error getting master registration for 127.8.204.124:43863: OK, instance_id { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" instance_seqno: 1786060896054154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 00:01:36.142102 8234 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 00:01:36.143008 8486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 00:01:36.144059 8486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 00:01:36.147799 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.131164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.8.204.125:37705)
0807 00:01:36.131284 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.131289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.133509 (+ 2220us) server_negotiation.cc:407] Connection header received
0807 00:01:36.133744 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.133748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.133798 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.133859 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.134973 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.135723 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.136862 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.137173 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.137252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.144086 (+ 6834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.144090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.144093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.144118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.145838 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.146327 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.146332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.146334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.146429 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.146798 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.146803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.146806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.147037 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.147145 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.147515 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.147636 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 00:01:36.157564 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.132189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.8.204.124:43863)
0807 00:01:36.132311 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.132316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.138950 (+ 6634us) server_negotiation.cc:407] Connection header received
0807 00:01:36.139076 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.139081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.139130 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.139223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.144359 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.145204 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.150853 (+ 5649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.151179 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.151263 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.153945 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.153949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.153952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.153992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.155732 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.156228 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.156231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.156233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.156314 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.156599 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.156602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.156604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.156816 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.156926 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.157280 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.157400 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 00:01:36.163470 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:36.132322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.8.204.126:37491)
0807 00:01:36.132608 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.132613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.139250 (+ 6637us) server_negotiation.cc:407] Connection header received
0807 00:01:36.139426 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.139429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.139495 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.139555 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.145358 (+ 5803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.146116 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.151407 (+ 5291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:36.151714 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:36.151792 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:36.159524 (+ 7732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 00:01:36.159528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 00:01:36.159531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 00:01:36.159561 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 00:01:36.161262 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.161735 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.161738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.161740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.161827 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 00:01:36.162134 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 00:01:36.162137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 00:01:36.162139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 00:01:36.162355 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 00:01:36.162472 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:36.163150 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 00:01:36.163276 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260807 00:01:36.164054 8399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [term 1 FOLLOWER]: Refusing update from remote peer 3c06941586994d82992da5272faec75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:36.164749 8445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:36.165434 8290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [term 1 FOLLOWER]: Refusing update from remote peer 3c06941586994d82992da5272faec75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 00:01:36.166940 8445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 00:01:36.169238 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c06941586994d82992da5272faec75c. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.169378 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.172585 8445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.172722 8445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:36.173885 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c06941586994d82992da5272faec75c. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.174016 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.174141 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.174263 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.178319 8517 mvcc.cc:205] Tried to move back new op lower bound from 7315705430641070080 to 7315705430539022336. Current Snapshot: MvccSnapshot[applied={T|T < 7315705430641070080}]
I20260807 00:01:36.180187 8486 catalog_manager.cc:1347] Generated new cluster ID: c61dfdc61d264da5b07e94af7c8275f4
I20260807 00:01:36.180258 8486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 00:01:36.203518 8481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.203701 8481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c [sys.catalog]: This master's current role is: LEADER
I20260807 00:01:36.206789 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c06941586994d82992da5272faec75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c06941586994d82992da5272faec75c" member_type: VOTER last_known_addr { host: "127.8.204.126" port: 37491 } } peers { permanent_uuid: "a3bad300f5ef4bbaaf20d86cf9aeb79a" member_type: VOTER last_known_addr { host: "127.8.204.125" port: 37705 } } peers { permanent_uuid: "3db68f4b9a3840a4bd8824fee4e83ed4" member_type: VOTER last_known_addr { host: "127.8.204.124" port: 43863 } } }
I20260807 00:01:36.206914 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 00:01:36.207466 8520 mvcc.cc:205] Tried to move back new op lower bound from 7315705430641070080 to 7315705430539022336. Current Snapshot: MvccSnapshot[applied={T|T < 7315705430641070080}]
I20260807 00:01:36.213376 8486 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 00:01:36.216718 8486 catalog_manager.cc:1504] Loading token signing keys...
I20260807 00:01:36.233883 8486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c06941586994d82992da5272faec75c: Generated new TSK 0
I20260807 00:01:36.234750 8486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 00:01:36.266415 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.263134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.8.204.125:37705)
0807 00:01:36.263270 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.263275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.265240 (+ 1965us) server_negotiation.cc:407] Connection header received
0807 00:01:36.265443 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.265447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.265515 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.265572 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.266273 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51126: BlockingRecv error: recv got EOF from 127.0.0.1:51126 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 00:01:36.268172 8231 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.264624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.8.204.126:37491)
0807 00:01:36.264747 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.264751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.267173 (+ 2422us) server_negotiation.cc:407] Connection header received
0807 00:01:36.267433 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.267437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.267501 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.267552 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.268053 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52662: BlockingRecv error: recv got EOF from 127.0.0.1:52662 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 00:01:36.269224 8439 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.263971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.8.204.124:43863)
0807 00:01:36.264088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.264093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.267000 (+ 2907us) server_negotiation.cc:407] Connection header received
0807 00:01:36.268444 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.268448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.268516 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.268570 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.269106 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45126: BlockingRecv error: recv got EOF from 127.0.0.1:45126 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260807 00:01:36.273974 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.272393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.8.204.125:37705)
0807 00:01:36.272507 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.272511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.273005 (+ 494us) server_negotiation.cc:407] Connection header received
0807 00:01:36.273224 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.273239 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.273311 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.273361 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.273872 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51138: BlockingRecv error: recv got EOF from 127.0.0.1:51138 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260807 00:01:36.278604 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.277185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.8.204.125:37705)
0807 00:01:36.277278 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.277282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.277799 (+ 517us) server_negotiation.cc:407] Connection header received
0807 00:01:36.277966 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.277969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.278015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.278062 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.278517 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51144: BlockingRecv error: recv got EOF from 127.0.0.1:51144 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260807 00:01:36.283854 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.282450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.8.204.125:37705)
0807 00:01:36.282551 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.282555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.283059 (+ 504us) server_negotiation.cc:407] Connection header received
0807 00:01:36.283222 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.283225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.283269 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.283315 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.283764 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51150: BlockingRecv error: recv got EOF from 127.0.0.1:51150 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 00:01:36.291235 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.289673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.8.204.125:37705)
0807 00:01:36.289793 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.289797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.290307 (+ 510us) server_negotiation.cc:407] Connection header received
0807 00:01:36.290510 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.290513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.290562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.290614 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.291143 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51152: BlockingRecv error: recv got EOF from 127.0.0.1:51152 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 00:01:36.303905 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.302136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.8.204.125:37705)
0807 00:01:36.302241 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.302246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.302891 (+ 645us) server_negotiation.cc:407] Connection header received
0807 00:01:36.303112 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.303116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.303167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.303224 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.303799 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51168: BlockingRecv error: recv got EOF from 127.0.0.1:51168 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 00:01:36.324370 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.322416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.8.204.125:37705)
0807 00:01:36.322565 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.322570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.323322 (+ 752us) server_negotiation.cc:407] Connection header received
0807 00:01:36.323558 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.323562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.323618 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.323671 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.324249 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51176: BlockingRecv error: recv got EOF from 127.0.0.1:51176 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 00:01:36.360294 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.358952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.8.204.125:37705)
0807 00:01:36.359120 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.359124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.359372 (+ 248us) server_negotiation.cc:407] Connection header received
0807 00:01:36.359559 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.359563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.359623 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.359715 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.360178 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51188: BlockingRecv error: recv got EOF from 127.0.0.1:51188 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 00:01:36.427738 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.426554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.8.204.125:37705)
0807 00:01:36.426754 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.426758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.426859 (+ 101us) server_negotiation.cc:407] Connection header received
0807 00:01:36.426990 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.426992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.427045 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.427126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.427602 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51190: BlockingRecv error: recv got EOF from 127.0.0.1:51190 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 00:01:36.559167 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.557864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.8.204.125:37705)
0807 00:01:36.558022 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.558027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.558225 (+ 198us) server_negotiation.cc:407] Connection header received
0807 00:01:36.558410 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.558418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.558469 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.558565 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.559063 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51200: BlockingRecv error: recv got EOF from 127.0.0.1:51200 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 00:01:36.818600 8337 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 00:01:36.817471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.8.204.125:37705)
0807 00:01:36.817602 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:36.817607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:36.817625 (+ 18us) server_negotiation.cc:407] Connection header received
0807 00:01:36.817818 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:36.817821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:36.817923 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:36.818021 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 00:01:36.818497 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51216: BlockingRecv error: recv got EOF from 127.0.0.1:51216 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 00:01:37.129460 8480 catalog_manager.cc:1259] Loaded cluster ID: c61dfdc61d264da5b07e94af7c8275f4
I20260807 00:01:37.129598 8480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: loading cluster ID for follower catalog manager: success
I20260807 00:01:37.131423 8480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: acquiring CA information for follower catalog manager: success
I20260807 00:01:37.132797 8480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3bad300f5ef4bbaaf20d86cf9aeb79a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 00:01:37.139160 8511 catalog_manager.cc:1259] Loaded cluster ID: c61dfdc61d264da5b07e94af7c8275f4
I20260807 00:01:37.139225 8511 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: loading cluster ID for follower catalog manager: success
I20260807 00:01:37.140753 8511 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: acquiring CA information for follower catalog manager: success
I20260807 00:01:37.141831 8511 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3db68f4b9a3840a4bd8824fee4e83ed4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 00:01:37.337078 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:37.332824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.8.204.125:37705)
0807 00:01:37.333127 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:37.333133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:37.333231 (+ 98us) server_negotiation.cc:407] Connection header received
0807 00:01:37.333423 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:37.333433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:37.333568 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:37.333698 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:37.333976 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.334507 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.335479 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.335714 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.335824 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:37.336169 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:37.336638 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:37.336772 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:37.336850 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 00:01:37.336906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260807 00:01:37.343784 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:37.340076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.8.204.124:43863)
0807 00:01:37.340320 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:37.340324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:37.340338 (+ 14us) server_negotiation.cc:407] Connection header received
0807 00:01:37.340479 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:37.340481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:37.340592 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:37.340716 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:37.340923 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.341467 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.342284 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.342528 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.342642 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:37.342964 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:37.343376 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:37.343509 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:37.343576 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 00:01:37.343628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260807 00:01:37.350524 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:37.346638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.8.204.125:37705)
0807 00:01:37.346807 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:37.346811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:37.347272 (+ 461us) server_negotiation.cc:407] Connection header received
0807 00:01:37.347360 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:37.347364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:37.347411 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:37.347488 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:37.347801 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.348417 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.349352 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.349650 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.349753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:37.350024 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:37.350100 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:37.350169 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:37.350327 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 00:01:37.350393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 00:01:37.351110 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:37.347039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.8.204.124:43863)
0807 00:01:37.347149 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:37.347152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:37.347165 (+ 13us) server_negotiation.cc:407] Connection header received
0807 00:01:37.347202 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:37.347204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:37.347268 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:37.347352 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:37.347719 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.348413 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.349678 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.350045 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.350153 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:37.350443 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:37.350527 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:37.350578 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:37.350898 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 00:01:37.350948 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 00:01:37.352202 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 00:01:37.346686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.8.204.126:37491)
0807 00:01:37.347284 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0807 00:01:37.347289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 00:01:37.347525 (+ 236us) server_negotiation.cc:407] Connection header received
0807 00:01:37.347862 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 00:01:37.347866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 00:01:37.348016 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 00:01:37.348142 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 00:01:37.348609 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.349445 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.350579 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 00:01:37.350935 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 00:01:37.351029 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 00:01:37.351441 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 00:01:37.351776 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 00:01:37.351866 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 00:01:37.352040 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 00:01:37.352097 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":78,"threads_started":1}
I20260807 00:01:37.355437 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 8127
I20260807 00:01:37.370971 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 8234
I20260807 00:01:37.387624 9009 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJxSMk/build/debug/bin/kudu with pid 8338
2026-08-07T00:01:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3290 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6583 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288149 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST